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' I. INTRODUCTION 

a - ^ • 

In. 1972, Educational Testing Service under contract with the Office of 

c 

a 

Naval Research, \Personnel and Training Research Program, began work on 

. ■ ■ / * ^ 

the first phase of a concerted series to study 'and improve theory and 

practice ,of ttaining/education program evaluation. The first phase ^ 

culminated" in a book, published by Jossey-Bass entitled Encyclopedia of 

educational Evaluation .' This book presented in relatively, non-technical 

language major concepts and practices in the evaluation of training or 

educational programs. Its intended audiences' included naval personnel .who^ 

commissioned or conc^ihted such evaluations. In the first six months 

following its publication, more than 6,000 copies of the book were sold. It ' 

1 

is now in Its third printing. 

Two more phases of the project were subsequently carried out after 
the completion of phase one. The second phase of the project is reported 
here. It is a survey of actual evaluation practices in various kinds ot 
settings involving adult education/ training programs; and its rationale, 
methodology, results, and conclusions will be reported. in detail below. 

The third phase of the project, presented as a separate report, is a. 
codification of evaluation principles and. a framework for appropriate 
evaluation practices. That is, among other purposes, the third phase 
provides, for evaluators and those co.mmissioning evaluiitions, checklists' and 
tabular presentations enabling a systematic approach to evaluation to be taken 



*See Ball, S. and Anderson, S. B.^ Professional issues in the evaluation of 
education/ training programs . Technical Report No. 3 for Contract No. ^ 

N00014--72-C-9433, NR 154-357. October, 1975. 



'However,, the topics addressed in phase^three,. while partly based on the 
opinions 'expressed by a distinguished panel brought together at Educational 
Testing Service, were also based on ^problems exposed by this survey. 
® While the rationale for this survey included providing data for the 
third phase of the project, other important functions were also served in . 
their own rigbt. One was to obtain new knowledge. Th.^e has never. been a 
survey of evaluation practices across a broad .spectrum of adult education/ 
training programs; and the 'state of the evaluation art as presented in the 
Encyclopedia of Educational Evaluation is of little consequence if it is' 
not in fact ^practiced iri that form. . ' ^ 

' By analogy 'we might look at the field of medicine. We would conclude 
that there is a missing gap in the application of our medical, knowledge if 
patients suffering a particular disease typically succumb to it even though 

\ professors of medicine know how to treat it successfully . Perhaps these 

\ . ^ ' * ' * ' ^ 

professors are not communicating their knowledge properly.; perhap's the 

" "front line" medical doctors are not alert to new ideas; or perhaps there 
are problems of a contextual kind (administrative arrangement-s , financial ^ 
* difficulties) that preclude the applibation of what i's kno^ga to what is done 
M program 'evaluation we have accumulated, expertise', techniques, and 
•insights, that ought to be useful if properly applied. Program evaluators • 
should be'able to do so if they are adequately * trained and the environment 
in which they work is- conducive to their carrying out their professional 
mission. But do they? . ' ^ 

A related question is whether in evaluating programs, a wide spectrum 
of evaluation models is employed or whether the rather traditional 
summative-typie evaluation is primarily involved. " . 



The maj,or questiop for this survey remains: What are the actual 
practices in program evaluation in different settings? This was the 
(Question that motivated the activities to, be reported. 

To answer this question "two related research activities were 
undertaken. First a survey of some 200 adult education/ training programs 
was carried out. Then 15 of them were chosen as exemplary and became 
;the subject of a site visit and case study. The procedures, results, 
and conclusions are presented in the following sections of this report. 



• , 11.^ PROCEDURES 

la this sedtion of the fepo.rt the sampling, the instrumentation, and 
the data collection. procedure will be described in detail. 
vA, Sampling Procedures . . 

A sampling procedure cannot ije rationally determined without some 
■ ^ prior consideration of the population about which, we wish to 

.geneiraliz-a.. The proposal- for this study had stipulated adult, 
. technical education/ training programs as the area of interest and the 
evaluations of ^hese programs as the focus of investigation. 
"Preliminary eOort-s to obtain a census of these ^^^pro grams from^ which 
to draw a called for sample of 200 programs not o\ly proved to be 
fruitless blit also pointed 'up the need for further elaborations of the 
population. 'The major problem was th^t different degrees of technical 
sophistication and systT.matization existed among different adult 

technical training programs depending" largely on their source Sof 

♦ 

funding and author'ity.^ • 

Four" major groupings of adult technical, training programs were 
f - • ' * ■ ^ 

discerned,. . ^ " ' 

' ' • ^ / . • ^ ' 

a) Those provided by • the- Department of Defense through the Army, 

Navy,, Air Force, and ^Marines 

b) Those provided by federal government departments and agencies 
other than the Department of Defense. 

""c) Those provided by state* and local government agencies, usually, 
as we found out, through ,.j unior/community colleges, 
d) Those provided by the private sector of the economy by 

industrial/commercial organizations for employees and potential 
employees. 
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For only one of these fouv categories: were we able to obtain a 
complete program list^ing. It was possible to obtain a listing of all 
technical programs provided by the Department of Defense. The listing 
was voluminous running in all to thousands of specific courses. 
However, Che o tiler three groupings had no such listing. It seemed 
possible that their offerings were as vast as th^t of the Department 
of Defense— especially those 'provided by., state and local governments 

■ and agencies [see c) above]. Unfortunately^ there was no singl^^locus 

\ 

of inforination, \ Each agency, each junior college, each commercial \ 



\ 

\ 



^organ i zation had its own- listing but the numb er of agencies, 

organizations, and colleges, was so great as to 'preclude our establishing 
an overall population frame. _ ■ 

As -well as the problem of establishing a population' frame, there 
was the technical probilem of deciding On a sampling technique. We 
had already indicated that a cluster, sampling approach was used; and 
50 programs from each= of the four groupings- of programs listed above, 
were -selected. 



Because there was no population frame in three of the four groupings 
■ and "because the ^survey was essentially exploratory, it was neither .. 
essential nor possible to obtain a fully randomize^ sample. The _ 
decision was to develop within each cluster a large pool of programs 
that had preferred characteristics. Then we would randomly sample 
50 programs wichin each of those four pools. ' ' • 

""See^ for example: 1. Catalog of Na w Training: Courses (GANTRAG) , NAVTRA 
.^lOSOO, Naval Training Command. July, 1973. ' m 

2. U. S. Army Formal Schools Catalog , D.A.' Pamphlet 350-lU, 
-Variously dated from February, 1965 to February, 1973. 



The. major criteria for placement of a program into a pool was that 

if 

it be currently offered to adults, that it be technical in content 
(enrichment , cultural , and general educational courses did not qualify), 
and that it be first available within the past five years (if dates, 
could be ascertained). Where vast numbers of programs were readily 
available for selection there was no problem in getting a cross-section. 
Thus, in the. case *of Department of Defense programs, a cross-section of 
the four ^^^a/d forces was called for. Also in the case of the Department 
of Defense, programs involving security qualifications were not included. 
The obtaining of a pool of programs from federal government departments 
and agencies was carried.vout by calling departments and agencies randomly 



selected from a full listing until the requisite number of programs was 
obtained. For obtaihing a pool of state and local government initiated 
prdgTrams , ten states were randomly selected. Then county and/or local 
goveromeat agencies were contacted as well as state departments and agencies 
within those states -until the requisite number of programs was obtained. 
Indus trial/commerical programs were obtained by randomly selecting from 
within the top ^ 500 corporations and making enquiries until the requisite 



number of programs was obtained (no more than one per corporation). 

To save time and cost, all initial contacts were made by telephone. 
Each pool contained 200 programs from which an initial 50 were randomly 
selected. A second group of 50 was also selected to serve as replacements. 
Thus, at the end of the sampling pro,cess there was a listing for each of 
the four program categor^.^s (Department of Defense, etc.). of 50 programs 
and a back-up listing of ^S^j)rograms . 



Persons over 17 years of age was the definition of adult. 

The preference was for cecently developed programs because they would more 
likely have had some form of evaluation and we were interested in obtaining 
a current picture of evaluation practices. 

Q^^ere was subsequently a follow-up letter. (See Agpendix B) ■ 



The actual data gathering. process is • described belov in Section C, 
The sampling process described here and used in this survey ensured that 
a wide range and variety of programs were selected encompassing all major 
geographic ari'as of the United States and encompassing all major sources 
of adult technical training outside the conventional university and college 
baccalaureate and higher degree programs. 

A second part of the study called for a closer investigation of 

o 

15 exemplary program evaluations to take place after the initial survey. 
The 15 were chosen by a subjective process after the project staff had 
scrutinized the survey responses. Again, selection was broken down to 

insure that each of our four major categories of programs had at least 

three case study visits and to make sure adequate 'geographic representation 

was maintained overall. 

Instrumentation 

In order to focus on what we deemed specific areas of importance, 
we queried responsible program personnel using the questionnaire which 
appears in Appendix A. -Its first par-t sought information about the 
program. There -were 18 questions concerning the program's goa,ls, 
target audience, length, level, instructional methodology, and student 
evaluation practices. The second part of the questionnaire contained 
questions seeking detaired data on program evaluation activities.';. /There 
were 30 questions concerning formal or informal, formative or summative 
evaluation activities including questions' 'on. the source of the evaluation* 
funding,, the extent and depth of the evaluation, the measures and . 
evaluation- design, the kinds of analyses used, and the type of reporting 
carried out. 



As well, for the 15 program evaluations chasen for more intensive 
study, an open-ended set of 14 topics to be covered was developed. These 
" re: i 

1. What is the program like? (Audience size, initiator, student body, 
serving population, length, goals, how students are evaluated.) 

2. What kind of evaluation (s) were carried. out ? (What model, what 
were the evaluation's goals? What were its values?) 

3. Who carried it out ? (Why the evaluator was selected, .qualifications, 
experience, in- or out*-house, etc.) 

4. .Who paid for the evaluation? (How, under what circumstances, etc.) . 

5. Who were the audiences of the evaluation ? (Program f under s, 
.developers, potential students, etc.) 

6. What were-^the expected outcomes and benefits of the evaluation ? (If 
you can find out directly or by inference.) 

7. What did the evaluation (s) comprise of ? (Describe the evaluation (s) 
briefly. Processes and conclusions.) 

8 . Was. the evaluation' carried out with technical competence ? If not, 
what vent wrong ? (Look especially at variables, measures, design.) 

9. Did the evaluation (s) • accomplish the evaluation goals? (In whose 
opinion? 

10 . Did the evaluation seem to help the audience for ^whom. it was intended ? 

i. - ■ — - ' " 

Were any changes made as a result of the evaluation ? 

11 . Did the evaluation look fot unintended outcomes (program side effects) ? 
■12.' What was good about th^ evaluation (s) ? Was there anything quite 

noteworthy ? 

. ^ 8 

■ • " i . . 

■ 16 , i 



13. What were problems with the evaluation (s) ? How could they have 
been overcome ? (In whose opinion? Try to get different viewpoints.) 

14. ' Are there special lessons to be learned fron studying this 

evaluation ? (Be general.) 
C» Data Collection Procedure . 

In the first section of this chapter (Section A) the sampling 
procedure was described in detail. The' data collection began with a 
sample of 200 programs divided equally among our four categories . There 
was, as well, a similar back-up set of 200 programs. The reason for . 
this second set of programs can be understood by reference to the focus 
of the. survey. While adult, technical training programs was the area 

^ . ■ : > 

of concern, their evaluations were the focus of- the survey. A program 
which had no formal or informal evaluation of any kind (broadly defined) 
was not used in the actual surve.y and a replacement from the back-up 
set was obtained. 

The data collection process began- with a letter to the director or 
person in charge of the program ^see Appendix C) . A .yeek after the 
letter was sent, a telephone call was made to determine whether the prograA' 
had been the subject of any kind of ^evaluation (formal or informal, past 
or present). If the program director indicated that no evaluation 
existed, the program was dropped from the study and. a program from 
the back-up list was substituted. In this manner, 200 programs with 
some form of evaluation were available for further use in the survey. 
A first 'empirical indication of the heterogeneity of the four categories 
can be seen from their acceptance rates: 

, . .Almost all (90 percent) Department of Defense and private sector 
programs were reported to .have some form of evaluation. 
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. . .It was difficult to obtain this information for other federal 

government programs because with many there was a heavy degree of 
decentralization. An adult, technical training program, ostensibly 
administered by a Washington, D.C. project monitor might actually 
be run, in a number of centers each having some autonomy with 
respect to such questions as evaluation. An acce'ptance rate of 
72 percent probably provides an underestimate of the amount of • 
evaluation occurring with federal government adult technical 
training programs. 

. . .State and local governments and agencies represented the. lowest \ . 

acceptance level. Less than half of the initial, listing (46 percent) 
indicated the presence of some form of evaluation and^^all but two 
of the back-up listing of programs had to beused to obtain the 
required number o'f 50 programs vji^feh some form of evaluation. \^ 
Given, the final listing of '>200 programs with evaluations .(50 per 
category), it was now possible to go ahead with the major data collection. 

The original intention for the data collection was that it be 
carried out by telephone interview. .However, pre-survey, pilot work 
in the data collection procedure indicated that this would not prove - 
satisfactory. Some of the programs had multiple staffing, so" that while 
one member of the staff was best -qualified to answer questions about 
the program, other members of Staff were best qualified to answer 
questions about the program's evaluation. As well, at least some of 
the questions required searching of records or time for consideration 
of the best response. Therefore, .a -differ'ent approach was taken, the 
telephone ^interview technique being dropped and a mailed questionnaire 
substituted. -Two telephone follow-up calls were used, if needed. 

10 ' ' . 

18' . 



The case studies were conducted after the questionnaire survey 
was completed. As was indicated above under Sampling, the programs 
selected for further study were subjectively chosen for their exemplary 
evaluations. Some objective criteria based on the questionnaire 
responses were used 'in reaching that judgment. A program^ s evaluation 
was deemed exemplary if it was formally conducted, had some level of 
statistical analysis performed on data generated by the evaluation, 
and had been instrumental in improving programs. In two instances 
exceptions were made'. In one instance, the questionnaire response 
seemed to suggest an innovative evaluation had been performed and, 
in the other instance, an evaluation (while informal and pedestrian) 
was representative of others in that category of programs. 
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• III. RESULTS ' 

The Questionnaire Survey 

In the first two chapters of this report the r^ational-e and procedures 
of this study were presented. In order to find out what practices are 
occurring" with respect to the evaluation of adult, technical education/ 
training programs, 200 programs (50 in each of four categories) were 
surveyed using a mailed questionnaire.- It had already been established 
that , these programs had been subject to some form of evaluation. • 

An overall response rate to the mailed questionnaires of 70 percent 
was achieved. After two telephone call-backs, 142 of the questionnaires 
had been returned. The response rate for each of the four categories 
of programs was: 

. . .Department of Defense (84 percent return) 

. . .Other federal government departments and agencies (66 percent return) 
. . .Private business and industry (80 percent return) 

. . .State and local government departments and agencies (54 percent return) 
Thi3 response rate corresponds rather closely to data presenfced-^earlier 
in "this report on the presence of evalua^Lional processes in adult, 
technical trad^JLiig-prog'rams . It was seen in the data on tHe presence . • 
'^oi some form of evaluation (Section C of Chapter III) that Department of 
Defense programs had the .highest proportion of programs with evaluations 
and State and local government departments and agencies the lowest. It 
may be seen that responses to mailed questionnaires were also similarly 
ranked. A possible reason for this is that the former grouping being 
more likely to have evaluated their programs was also ^more inclined to 
answer questions about -them. ^ ■ , , 

A follow-up to determine causes of failures to respond was conducted. 
Twency-one of the 58 non-respondents (36.2 percent) were telephoned 'and 

u20 



reasons for non-response requested. The reasons provided were: 

. .Overlooked — will return it soon (4) ^ ■ . • 

. .Never received it (3) ' - 

. .Did return it — must have been lost in mail (3) 
.Been too busy (3) 

/'.Sent it on to colleague/assistant to fill in — didn^t get it back (2) 
. .Questionnaire too general or too awesome (2) 
. .Been ill or on maternity leave (2) ' - 

. .Cannot recall what happened (1) 

. .Person to whom sent no longer employed there (1) 
It would seem that the 29 percent non-response rate was not caused by 
systematic factors related to the nature or style of the questionnaire. 

The data from the l42 questionnaires were processed.. The major 
analyses called for were ^f requency counts and percentages for each response 
category in each of the questionnaire items. These analyses were 
conducted for all l42 respondent questionnaires overall and separately by 
each of the four program categories— Department of Defense (D.O.D.), . 
Other Federal Government Departments and Agencies (O.F.D.), State and 
vLocal Government Departments and Agencies (S,.5t L.), and. Priyate Business,' 
• Commerce, and Industrial Organizations (Industry). The results are so 
presented In Tables 1-5 at the end of this chapter. 

^•Then studj^ing these tables it is important to bear in mind the 
•nature of the respondent sample. The programs that are described (see . 
T.able 1) are adult, techni^cal training programs that initial enquiries 
indicated had been subjected to\evaluation. Therefore, "the results in this 



*Many of these programs are conducted by juniorXand community colleges, 

' ' 21 
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study refer to such programs and not to adult training programs in 
general. Furthermore, in considering the information about evaluation 
practices, it should be noted that t"here are many programs that do 
not have any form of evaluation. Evaluated training programs in 
relation to all programs probably range from a minority in the case 
of state and local government departments and agencies to about 
9CL-p.ercent in the case of Department of Defense programs. (See Chapter' 
II, Section C. ) . .. 

With this understanding of the nature, of the sample in mind, it 
is appropriate to consider Tables 1-5. 

Table 1 provides a description of the 142 programs from .which the • 
evaluation data in succeeding tables are based i It may be noted that: 
. . .The majority of the programs^as called for iu' the sampling^were 

initiated since 1968 (72.4 percent). The major source of variance 
was with the state and local gropu (mainly the junior/community 
. college programs) where 38.7 percent "of respondent . programs had 
begun before 1969. > ^ . " . 

• . .The median number of- students trained in these programs was* between 
150 and 199. The arithmetic mean was considerably higher because 
of the skewed distribution: About 20 percent of the programs had 
each trained over a thousand students. 
. .f.^.The successful completion rate for trainees is typically over 

80 percent. ' However, 17 of the l42vprograms reported that less 
than 50 percent of the trainees complete the program successfully. 
S ta te and local and private commercial and industrial programs 
accounted for 14 of these 17 programs. Heavy attrition rates seem- 
to be very 'rare with Federal Government (including Department of 
Defense) programs. 
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.Repeating of programs (QA2b. and QA2b2.) is possible in about a 
quarter of the programs. But it seems that few students avail 
themselves of this opportunity. 

.A third of the ''programs are offered in more than one location. 
Of that third, almost three quarters were in many (more than 
four) centers. The exception was in state and local government 
programs (mainly in j unior/community colleges) where the large 
majority of programs were presented at a single site, 

.Most of the state and local programs (56.7 percent) are accredited 
by some board or agency. However, only half the Department of 
Defense, 25 percent of the private, commercial/industrial and 
18 percent of the other federal department programs are accredited. 

.There is, similarly, considerable variation with respect to the 
payment of fees by students — from less than five percent with- 
Bepartment of Defense programs to 73.1 percent with state and 
local government and agency programs. 

.Almost all of the programs (94.4 percent) have a written statement 
of goals. Most (76.6 percent) have goals written in behavioral 
terms'; ' , ■ " ■ , ' 

.Formal prerequisites for trainees (QA9.) are not usually required 
for federal government programs but are for most- of the other 
programs. Only with state and local government programs does the 
prerequisite involve some educational level of attainment (probably 
the j unior/community college influence).. (See QAIO.) For the 
other 'three categories of programs the prerequisite involved some 
previous course or -skill acquisition. 



, . .Most programs are of less than four months duration, (QAll .) • 

Again the exception of the state and local governments category 
of programs is due to .tlie±rL_hB±ag-. administered through junior/ 
community colleges. The number of hours of outside classroom 
work (QA13.) varies greatly from less than' 10 hours- (31.9 percent) 
to more than 160 hours (23.9 percent), 

. . .Methods of instruction are varied and many (QA14.). Most frequently 
mentioned vere formal classroom teaching, practical laboratory/ 
• workshop experience, arid multimedia usag-e.' However, the question 

.V. 

asking the respondent to indicate the most important method of 
instruction achieved consensus with only formal classroom tea.ching 
and " practical laboratory/workshop experience receiving' more than 
ten percent endorsement. 
. . .The final series of questions concerning . the programs themselves 
enquired about student evaluation.^ In the majority of programs 
(78.0 percent) students' are formally evaluated (QA16.). As one 
might hope with adult, technical training programs, *the kin;' of 
formal student .evaluation carried out (QA18.) frequently includes 
performance (practical proficiency) examinations (54.2 percent) 
and instructor ratings of performance or products (46.5 percent). 
Performance (practical proficiency) examinations were most 
' frequently reported as the "most important elementV in the student 
*' evaluation. 

. In general. Table 1 provides a clear illustration of the quite 

large diversity among groups of adult, technical training programs, 

. /■ • 

depending in part upon ^t-heir source of funding and authority!. For 
example, most Department of Defense programs are formally "evaluated, 

. /;'• • ' 2^ ■ ■ 
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have low attrition rate.s, and do not charge student fees. However, 
state and locally sponsored programs often have no evaluation component, 
frequently charge student fees, are usually available at only one site, 
and m^^ involve h^-avy student attrition rates. A more comprehensive 
feelz.ng for the diversity of the programs being considered here is 
provMed in Section B of this chapter when in-depth case s'tudies of 
programs^are presented. 

Of course, the reaso^^for describing the programs is primarily 
to provide a backdrop against which to understand what is occurring on 
the stage of program evaluation. Table 2 presents information on the 
number and percentage of the respondent programs that have been subject 
to formal evaluation.- (Formal was defined as some conscious, planned 
effort to provide-an evaluation. Level of evaluation sophisitication 
-w^s-^n£>t--a--f-aator in this, def inition. )_ 

Overall, 69.0 percent of the programs had been formally evaluated 
for the purpose*^ of improving the program (18.3 percent), assess the 
jijnpact of the program (2.1 percent), or for both these reasons 
(48.6 percent). Note that program improvement predominated over 
program impact ^assessment as a justification for the evaluation. - 

Table 3 provides inf orijiation on the formal evaluations of the 
respondent programs. The descriptive statis'tics are presented 
separately for the improvement and the impact elements of, the 
evaluations. From Table 3 it can be seen > that:. 

. . .Program administration and the program' s educational staff were 
primarily responsible for calling for the evaluation. 'Outside 
agencies and other factors have little influence except in the case 



of the Department of Defense prog^rams where j, as we -shall see latei; 
in the case studies (Section B of this chapter), there is' likely 
to be an evaJ^dation pjrocess mandated by regulations. 

.The evaluation is usually carried out by -'insiders^' — program 
administrators, 'developers, or educational staff. Again an 
exception is the Department of Defense where specializations and 
division of labor not infrequently , provides for external 
evaluation. . . ' . . 

.Internal funding of evaluations (see QB5 . ) is the rule across all 
categories. ^ ^ ^ . . 

.Almost all the evaluations reported quite^ recent evaluations (past 
two -years). The length of the evaluation ranges widely from 
less than a week to over one year. . Note :that 41.9- percent indicated 
a continual evaluation -process. The continual process- is ^ 
especially apparent with Department of Defense and other federal 
government programs . ^Hlt is also more likely to be adopted for 
improvement than for impact purposes. 

.The focus of .measurement was m'ost frequently t'he students and the 
curriculum .(QBIO.) and the measurement techniques used were most/- 
frequently interviews* and questionnaires or observation scale"*s 
(QBll.). Program evaluation, using standardized or teacher made 
tests was relatively infrequent. This sugg^es-ts .that formal 
'program evaluations, as currently performed, are relatively 
unlikely to be concerned with program effects on trainee's assessed 
by the trainees' performance on paper and pencil tests. 

.The reliability and validity of t^he measures ^does not seem to be 

a major concern overall. In less than 50 percent of the cases were 
• ' . ''^^ 

they assessed. ^ o. 
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* .The measures that were used in the evaluations primarily assessed/ 
program goal areas and only secondarily assessed potential side-^ 
effects. k sligh't difference concerning side-eif f ects measures • 
can be noted between improvement studies (39.1 percent included 
such measures) and impact studies (50.7 percent included such 

studies)* \ 
, •Mo^t cx the» evaluation designs eschewed the use of the true 

experiment or quasi-experimeqt. The picture that emerges is that 
the typical program evaluation is an observational (survey) ^ study 
using, questio^nnaires and interviews but not using statistical 

analyses (see QB18.). \ ' 

. About half the evaluations prodvlc-ed a written^ report (QB20.) but 
when we asked could we have a. copy only ten ptositive f^sponses . 
were received (improvement studies) and" nine positive responses 
(impact studies)* ■ Even assuming that , ^ for ' some progra^QS, 
coixfidentiality -requirements precluded our being offered a copy . 
of the evaluation report, their relative unavailability evokes 
some suspicion about their ^finished qual-ity. ' * . 
, . .-One pleasant feature of |he 'responses was the rather frequent 

claimj||||k3 percent) that the evaluation promot^ed program changes 
^ Even wixn impact studies, 60.. 6 percent claimed they were change- 
inducing. . „ 5/" 
In^general, Table 3 indicates that formal evaluations of adult, ^ 
technical training education programs typically • involve questionnaires 
and/or interviews of trainees and teachers. The^ study mainly intended 
goal areas of the program. They are usually called fox, funded, and 
carried out by insiders rather than agencies external . to the program. 
Most evaluations seem "to engender program change; but few seem to resul 
in- an, aiaailabia written report. ' ^ 
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The final section'of the questionnaire on program evaluations dealt 
with evaluations deemed to be informal. This group of evaluations i- 
constituted 41 of the 142 program evaluations studied indicating that 
even.. when programs have evaluations, about one--quarter of them are of 
the unplanned variety. The specific tallies showing the purpose to 
which -these informal evaluations are put'^are presented in' Table 4. 
Note that, as for formal evaluations, program improvement predominates 
over program impact as the major purpose. 

Table 5 presents further informat^ion about ,the 41 iiiformal 
e\^aluations . Where the purpose was primarily improvement, instructors 
and administrators c^rry out the evaluation. This is also true of . ' 

the impact studies except that there a 'higher ^proportion of the cases 
indi/cated -vthe presence- of some third party. Presumably, impact studies 
call for a greater degree of objectivity than improvement, studies, 
and this- objectivity is evidenced by the use lOf a person or agency 
other than the program's instructors or administrators. 

^ . The other question put. to respondents concerning informal evaluations 
asked what information was used to guide subsequent program improvement. 
'(Note that only one"-of the 41- informal evaluations did not have 
program ^improvement as a purpose.) Of the 40 responses to this 
question, almost all '(92 . 5' percent) mentioned instructors judgments, 

70.0 percent men-t-ioned students' judgments, and 62.5 percent mentioned 

. . ' ' J" . 

the ' j'ud^gments of other observer&'such as administrators. That is, 
informal, unplanned evaluations, probably perforce^ rely heavily .on . 
the ^ judgments of the major pro-grapa participants in ef f ecting program 
improvement". . .. 

.20 , ■ ' 
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t:. Questions concerning informal evaluations were ^purposely kept few. 
Informal evaluations represent the twilight between the daylight of 
planned program evaluation and the darkness of no evaluation. True to 
expectation these unplanned, informal evaluations were carried out 
mainly by program staff assessing the judgments of program staff 
presumably in simple ways. To generalize from our sample, this 
unsophisticated approach probably represents about a quarter of all 
program evaluations, . " • 

B, .' The Site Visits 

As well as the data obtained from the 142 questionnaire respondents, 
the study called for a more in-depth investigation -of a small group 
of 15 programs with exemplary evaluations. The criteria for selecting 
these programs have been presented in Section C, Chapter II, 

The ETS staff who made the site visits- were given a set of 14 major 
questions to be answered in their r-,site visit reports. These questions 
have been presented in Section B, Chapter II, 

Fourteen of the 15 site visits were conducted approximately on • 
schedule, The subsequent, reports were edited to retain anonymity of 
persons and programs and abridged to conserve space. The edited, 
abridged versions are presented in Appendix D, 

One of the 15 site visits was never completed. An initial visit 
to set up the ^*case study" took place. But subsequently, there was a 
long ,$eries of po;stponements due to illness, vacation, company 
reorganization, busy schedules, and eventually, as' we understand it, 
the termination df the contact person's employment with that ' company , 
The decision was 'made at that late stage not to draw a new program 
site for study, / ' 



For example, accompanying reports and forms were excluded, 



Appendix D provides a data base that indicates the variability 
of the-^l^^ programs and of their evaluations, (Incidentally, it also 
indicates the variability of the site visitors — a factor to be expected 
in case study reports.) We shall here provide a summary of the major 
results of the site visits by posing the qvics tions the ETS site visitors 
had in mind as thpy interviewed the program personnel. 
What is the program like? 

The major impression in looking;*over the case studies is one of 
great program diversity. This can perhaps best be shown by naming and 
briefly describing the programs in their order of .representation in 
Appendix D. , ' ' 

1. A drug abuse education specialist course — a sophisticated program 

to help solve a complex problem by developing a cadre of specialists 
with an intensive understanding of drug problems (and their ^ • ^ 
etiology) and social skills to work with drug addicts.. 

2. A digital subscriber terminal equipment repair course provided 
m^nly for seryicemen other than commissioned officers. Mostly 
"hands on'* training is involved and 15-20 weeks of intensive work 
is typically the duration. 



An air traffice 
per year are pr 



controllers training program. Up to 1000 trainees 
TDcessed to meet FAA requirements. A high level 
of electronic aptitucie is required of the enrollees. 
4. A radar repair course of 25, weeks duration. The course is for 

enlisted servicemen. very high level of electronic aptitude is 
required of thelenrollees . ' ■ 
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A management by obj ectives 'course for . second 'level supervisors 
in a federal government agency. The course is of three days 
duration spaced over about a 30-day period* 

Supervisory training - Phase 1 for a federal government department. 
A requirement f or ^all new supervisors, the course covers two 
.40-hour blocks of time and attempts to teach skills of how to 
deal effectively with people. 

A course on clear writing for personnel in a special federal 
government service. Attendance at t:he course ranges from 20 to 
60. , 

Executive development program for professional employees of a 

federal government department with low management or supervisory 
..experience. So far 120 trainees are in this new- course. 

Food and beverage management program for employees of a major 
■motel chain. Class size is 35 and the trainees come from 

throughout the USA. 

A sales fundamentals course conducted by the Division of industrial 
Relations of a large corporation. The course is for sales 
personnel new to the company. About 75 trainees per year attend 
the course. 

Managing the manufacturing operation is one of five autonomous 
units presented each quarter by "the Education and Training Division 
of a large corporation. . The course is for management level 
employees or those with the potential- for work at this level. 
Medical laboratory technician program carried out in a community " 
college. A considerable amount of on-the-job training is involved* 
in the program. ^ ' 
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13. Environmental aide program run by a coynty area vocational education'' 
center to train assistants. The program provides experience so 

the trainees can be later employed in such occupational areas as 
sewage and water treatment, soil conservation, and air pollution 
control, 

14. Plastics technology program offered at a technical institute which 
is part of a state vocational technical and adult education 
network. 

Note that the programs range in scope, from machinery repairs to 
higher level management, in^leng'th from a few days to two years, and 
in- extent from a few students in one location to thousands of students 
in many locations. 

The kind of evaluation ? carried it out ? ' Wlio paid for i t? 

The kind of program evaluation, as one might expect for a wide 
variety of programs, also varied widely. The first four case studies 
are of Department of Defense programs. There .is a clear sense o£ a 
systematic approach to the program evaluations with considerable evaluation 
expertise being brought in by "contracts ^ to outside consultants as the con- 
sultants were needed. Note, for example, in Case Study ir2.^ ^'Evaluation in t 
Axmy's schools,. . .is part 'of a comprehensive system, . .more formal, 
more regular, and institutionalized to a greate: extent than efforts 
elsewhere described. ..." Note, too, in Case *";udy //3^. the availability 
of a Program Evaluation Division. 

The other federal government program evaluation case studies (//5-//8) 
reveal a feeling of awareness of the value of evaluation but not the 
expertise to carry it out. It seems that in-housa evaluation is typical. 
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Note, for^ example, in Case Study //7: ''Evaluation of training is required 
by Civil Service regulations tut no guidelines are given fo^ the type or 
extent* . . .There is no' separate budget for evaluation activities.'* 
Or as another example consider the statement -in Case Study //8: ."For such 
a carefully planned and documented program with a great deal of emphasis 
on evaluation feedback to participants, there is surprisingly little planning' 
or execution of a program evaluation." 

The- evaluators for the programs conducted by private corporations 
(cpimnerce, business, and industry) imbued us with even less confidence. 
In at least one case the'^program evaluation was primarily a political 
exercise. Consider as ^a program evaluation, goal: "To show my boss how 
good the program is.'* - . 

State and local government and agency program evaluations seemed to 

vary .depending on whether the programs were conducted in junior/ community 

I 

colleges or not. We draw a little upon previous experience and the 
questionnaire survey for this statement. However, the. case studies provided 
a good illustration. The community college program was evaluated in a. 
rather traditional college way primarily by self-study. When the programs 
were conducted by vocational and technical institutions, a more -sophisticated 
approach was used. ' • ■ 

Who were ' the audiences of the evaluation ? \fhat were the expected outcomes 
and benefits ? Technical Competence ? Evaluation goals accomplished? Help 
the audience for whom intended ?- 

It makes sense in providing a' summary of the case studies to combine 
these questions (//5 to 7/10 in" Section B, Chapter II) for they integrate the 
major evaluation processes from planning to dissemination. 
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Again the pattern noted before continues to operate. Department of 
'Defense program evaluations seem to be the mpst systematically planned and 
their' results disseminated. Audiences were program staff and officers 
in charge; the maj'or benefits expected were improved curriculum and 
teaching methods. 

These terse remarks hardly mine the rich Iqde of comments and 
descriptions to be . found on the four Department of Defense case studies. 
HdweveTT^ many of these remarks are not , generalizable across programs. 
For exai!iple, the fourth case study points to a problem created by a 
'different staff group working on the evaluation than is involved in the 
program. But the resulting relative lack of coordination in the use of 
evaluation results is not a problem noted in the pther three case studies. 

Other federal goyemment program evaluations, as presented in* their 
case studies, are less systematically planned and executed. However, .-^ 
at least in case studies //5 and #6 there was evidence that an unsophisticated 
approach to evaluation did not prevent the results beneficially influencing 
the programs. 'With case studies #7 and //8"{, monitoring and staying within 
guidelines and regulations seemed to be two major evaluation functions 
and these were not conducive to having substantial influence on program 
improvement. 

The evaluations of -private commercial/business/industrial programs • 
seemed to be quite varied ranging from something resembling a caricature 
of an evaluation to something resembling a well-finished portrait. The, 
reader is invited to find the proper recipient of these labels. The same 
wide variation in quality may be noted in the state and local government 



This was true from an examination of the surVey results too. 
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'program evaluations. Studies ifl3 and //14 provide illustrations of 
thoughtful evaluation planning" and the establishment by controlling 
agencies of sensible guidelines. . ' . * 

What was noteworthy about the evaluations ? Were there problems ? Could 
they have been overcome ? Are there any special lessons to be learned 
fropi studying this evaluation ? • ^ 

It is not possible, though it would literally be fabulous, to pull 
together 14 diverse case studies and come up with the moral for program 
evaluation. Tlie best we can do in these circumstances is to list what 
seem to be the most important lessons and 'comments and then Took for some 
comraon elements. 

. . .Innovative and developmental programs probably need less restricted 

evaluation designs and models. 
.. . .Evaluations, regardless of model or design, constantly interact 

with. the nature of the personnel who are being assessed, receive the 

reports, or must make decisions. . ^ 

. . .Needs analysis prior to program development continues to be an often 

ignored aspect of the evaluation process. 
. . .The program staff becomes more open and cooperative when the;^ find 

the evaluatiu will be useful to them. , ^ 
^. . . .Everi,«w,hen inferential statistical analyses ^re missing, very useful 

evaluation outcomes occur. (This thou'ght was repeated more than once. 
. . .Training programs lend themselves to evaluation that can be . 

■ characterized as s^ystematic and where words such as "feedback" and 

"performance criteria" apply. Education programs^ on the other hand, * 

are more general in scope and, therefore, mare difficult to evaluate^ 



.When evaluators also know the subject matter of the program their 
credibility with program people rises. 

.If' the audience for whom the evaluation is 'intended does not; feel 
that the evaluation is important, little use will be made of , the 
findings. . ^ 

.Outcomes from learning opportunities for managers are difficult to 
document. Managers return to a great variety of tasks unlike' 
people who are being trained, say., to repair dishwashers, 

.It is unfortunate that a program that gives such attention to its 
training materials 'gives such scant attention to its evaluation. 
(This thought was also echoed in another case study.) 

.There is some indication that the evaluation efforts are improving. 
There is planning for a system of need assessment. 

.Private sector, commercial organizations usually need professional 
help from outside to develop adequate program evaluation procedures. 

.Adequate program evaluation requires adequate lead time. 

.You cannot- properly t^ll how much skill or knowledge the trainees 
gained by asking them to tell you on a questionnaire what their- 
perceptions were. 

.If the perceptions of program participants are regarded as important 
then all the various groups of par ticipan^s^trainees, teachers, 
administrators, supervisors back on the job), p^robably should be 
questioned. . \^ 

..Long-term follow-ups are rare but impoYtant. 
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. . i "The most obvious lesson that can be learned from this evaluation 
is the contribution that evaluation can make in helping assure an 
effective training program^" (See p. D-60 below.) 

-k-k-k-k-k^-k-k-k-k-k-k-k 

Of course, there is much that is professionally rewarding going on 
in ,the evaluations described in the case studies. Programs are being 
improve d, trainees and teachers are being he]^e4.,_an.d^a-^se^ pu-:^po&e- 



'can be seen being generated at least in some of the 14 studies. 

Whal is disconcerting in looking over the case studies is that although 
•they were\chosen as being exemplary illustrations of program evaluation 
based on reasonable' and objective criteria, the in-depth studies^ indicate 
many major deficiencies. We are not arguing that vast sums^ of money or 
great professional expez^tise should be coniiinually devoted to program 
evaluation. But^ we are arguing that minimal requirements demand some 
systematization of the evaluation 'process , some attention to such details 
as planning of the studies .and dissemination and utilization of the results. 
Departmen^t of Defense training programs seem to be closest to this 
reasonable requirement. The other three categories' programs seem to be 
evaluated well or poorly depending not on the institution they serve but 
on the professipnal quality of the particular individuals who happen to 
be entrusted with^ the relevant duties. This leads to a haphazard situation 

which cannot be recommended. ' . * 

■ ^ ■ 

kkki^kkkk-kk 
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IV. SUMMARY ANDv.CONCLUSIpNS 

In 1972, Educational Testing Service began work on a series of 
studies ant/ theoretical papers in' an attempt to improve the theory and 
practice of training program evaluation. .This report presents the second 
phase of the series — a survey of current evaluation practices across a 
broad spectrum of adult technical training programs. 

■ The survey consisted of two parts. First, a questionnaire survey 
of 200 program evaluations divided equally among four categories — Department 
of Defense, othet: xederaT"^oVefnlnenrt departments and agencies, state and 
local governments and agencies, including j unior/conimunity colleges, 
and private sector commercial, industrial, and business organizations. 

^. 

Obtaining the 200 programs with evaluations was not a simple task. ' 

, Except for the Department of Defense no broad listings of , programs could 

be obtained. We generated a large pool of programs in each of the four 

categories and randomly selected 50 (the initial list) and then another 

50 (the back-up list). If a program in the initial list was found to have 

been evaluated we retained the program. Otherwise,^ we eliminated it and 

tried instead a program from the back-up-list. Almost all -(90 percert) 

of 'the Department of Defense and private ' secLur programs had some form'. ) 

' - i 

of evaluation. Only 72 percent of other federal government programs and • 

■ 46 percent of state . and locals government programs in the initial listings 

had evaluations. • . ' 

A mailed questionnaire was sent to the program directors and an* 

overall response rate of 71 percent was achieved. The data from the survey 

were then analyzed and tabulated (see the final pages of Chapter III). 

^ As well as this questionnaire survey,' 14 case study site visits were 

conducted (Section B of Chapter III) in order to- obtain an in-depth 

^ picture of what is happening in exemplary program evaluations. 
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The 142 programs subject to questionnaire scrutiny were quite diverse, . 
not only among themselves but also between categories. Department of 
Defense programs, for example, rarely charge student fees. State- and 
local government programs are least likely rto be fofmally evaluated 
and least likely to, offer their programs in more than one center. .. 

Almost 70 percent of the programs had been formally evaluated usually , 
'for the purpose of program improvement rather than to assess program 
impact • The typical f oxxnaL-evaluat ion-^eonsis ts--"Of~-quest±onn 
interviews of trainees and teachers. They are usually called for, funded, 
and carried out by insiders rather than -by agencies external to the 
program. Writiten reports are rare; written reportg available to outsiders 
are even more/ rare. 

\Jhen^ thl evaluation is informal (unplanned) they consisted primarily 

r * ^ 

of program s/taff talking to program staff and participants in order to 
assess theii judgments; This non-recommencied approach was' taken by about 
a quarter of our 142 program evaluations. ' * 

Site Visits added to our conviction that adult, technical -training 
programs axe so vari'id in scope, length, and substance as to defy any kind 
of direct ^tiat^ on- how .they should be^ evaluated. From page 22 to the end 
^ of Chapter III presents a large range of insights obtained f.rom the site' 
visits. ^ ■ " 

One site visitor was told: "If this evaluation is exemplary then-' 

we're all in 'trouble . " Actually^ we found even here wide variation. Most 

% . ■ ■ 

' 0 

of the evaluations were indeed provxng helpful and were conducted with 
reasonable expertise.. Some, 'however , were not. The great need for a more 
systematic approach (except perhaps in the case of Departinent of Defense 
programs) was clear. Also clear was the relatively narrow interpretation 
of evaluation methods of investigation and the lack of adequate dissemination 
Q of evaluation results. ^ 
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V These atid related ' topics are the focus^/of. the third technical 



report in this evalua'tion study series, 

} 



*See Ball, S, and Anderson, S. B*. , Prpf esgional issues in' the evaluation of 
education/ training programs . . Technical Report ^o, 3 for Contract No. ■ ~ 
N00014-72-C--0433, NR 154-357. October^ 1975. " \ 4 
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TABLE 1 



Descriptive Information about Training Programs .as a Total| 
Group and Separately for Department of Defense ; (D. 0 .d'. ) , 
Other Federal Departments (O.F^D.), State and Local 
Gbverniaents and Agencies (S.&-,L.)> and Industry Subgroups 
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14.5 


3 


7.9 


3 


9.1 


^- 1 


3,7 


13 


32.5 


29 


21.0 


10 


26.3 


10 


30.3 " 


■. 0 


0.0 


9 


"22.5 


15 


10.9 


7 


18.4 


3 

7 


'9.1 


1 


3.7 


4 


10.0 



41 



33 



TABLE 1 (continued) 



QA2 , About how many student s/ traine*es have been trained ijwt;.llts^rograiii so. far? 





-. 'Total 


D,. 




r\ I? "n 


O • Of 


T 

Li • 


Industry 


■ ^ 


. 2 ' 


N 






N 


% 


■ N 


% 


0 - 4^9 


29 


22 . 7 


6 


18,2 


7 21.2 


■ "8 


32.0 


8 


21.6 


50 - 99 . y. . 




9.4 


3 


9.1 


1 3.0' 


4 


16.0 


4 ■ 


■io.s 


100 - 149 " 




11.7 


. 3. 


9.1 y 






8.0 


6 


16.2' 


15 


^--1^^.^12.1 


2 






150 - 199 


11 


8.6 


2 


6.|L 


2 0.1 


c 

J 




: 2 


5.4 


200 - 249 


8 


6.3 


3 


9J1 


3 J 9.1 


1 


lo 


1 ■■ 


2.7 


250 - 999 ■ 


30 


23.7 


9 


27 .p - 


P 24.2 


■ 4 


16.0 


-9 


24.3 


1000+ 


23 


18.2 


7 


21.1 


8 24.2 


1 


4.0 


7 


18.9 


Doa't know & omit 


14 




9 






2 




3 




■kit 

Mean and (SD) , 


** 




549,, 


756' 


* 


206 


(339) 


2097 


(6971) 



One program for which an enrollment of 4,000,000 students ovet l:h^ years created an 
unusually high mean^ and sfcandaTd deviation. . ♦ 



k-k 



V ANOVA analysis indicated no significant differences among the means, 
very high variances. 



Note, there were 



QA2a, What percentage of students/ trainees complete the program successfully? 



/ 

less than 50% 
51 - 60% 
61 - 70% . - 
71 - 80%. 
81 - 90% 
91 - 100% 



Total 


D 


.CD. 


0 


F.D.^ 


S.5. 


L. 


Industry 


N % 


N ■ % 


N % 


• n" '■ %. 


N % 


17 13.^4 


1 


2.6 


2 


6.5' 


7 


31.8' 


7 


20.0 


3 2.4 


0 


■ 0.0 


0 


0.0 


' 1 


, 4.5. 


2 


5.7 


■■2 • 1.6 


0 


0.0 


1 


3.2 


1 . 


4.5 


0 . 


0.0 


8 6.3 


4 


10.3 


1 


3 . 2' 


3 


13.6 


0 


0.0 


26 -20.5 


9 


23.1 


3 


9,7 


8 ' 


36.4 " 


6 


17.1 


71 55.9 


25 


64.1 


24 


'77.4 


2 


9.1 ^ 


20 


57.1 



34 
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TABLE 1 (continued) 



QA2b . "Can suecessful trainees repeat the , program? 





Total / 


D. 


O.D. 


0. 


F.D. 


S.& 


L. 


Industry 




N % 


N % 


N % 


N 7. 


■ N % 


Yes- 


91 


70.5 


17 


41.5 


25 


92.6 


25 


92.6 


24 


70.6 


No 


33 


25.6 


23 


56.1 


1 


3.7 


2 


7.4 


7 


20.6 


\^;V^pon't know 

V-' 


5 


'3.9 


1 


2.4 ■ 


1 


3.7 


0 


0.0 


3 


8.8 



QA2b2 . If so, (trainees can repeat) ,ii^t percentage of unsuccessful trainees repeat 
.the program? ■ , ' ' - . . 





Total 


D.O.D. 


O.F.D. 


S.& 


L. 


Industry 




N % 


N % 


N % 


N % 


N % 

C 


0% 


26 


42.6 


6 


42.9 


10 • 


66. "7 


5 


26.3 


5 


38.5 


1 - 20% 


26 


42.6 


3 


21.4 


4 


■ 26.7 


13 


6.° . 4 


6 


46.2 


21 - 40% 


1 


1.6 


1 


7.1 


0 


0.0 


0 


0.0 


0^ 


0.0 


41 - 60% 


\ 3 


4.9 


0 


0.0 


1 


-6.7 _ 


1 


5.3 


1 ' 


7.7 


61 - 80% 


0 


0.0 


0 


0.0 


0 


0.0 


0 


a.o 


0 


0.0 


81- 100% 


5 


8.2 


4 


28 . 6 


0 . 


0.0 


6 


" 0.0 


1 


7.7 






















\ 






















\ 


1 

Is this program offered in more than one center 


or location? 






t 




Total - 


-D.O.D. 


„ O.F.D. 


S. 


& L. 


Industry 




N 


% 


N % 


N- ' % 


N % 


N 


7. 


• Yes 


. 49 


34.5 


6 


, 14.3 


24 


72. 


4 ■ 


14.8 


15 


37.5 




93 


65.5 


36 


85.7 


s 

9 


27.3 


23 


85.2 


25 


62.5 


Don ' t. know 


0 


,, 0 . 0 


0 


0.0 


0 


0.0 


0 


0.0 


■ 0 


0.0 



ERIC 
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QA3b. 



, • . '. TABIDS 1 ^ (continued) 

If-so, in how many other centers or locations is it offered? 





Total 


■ D.O.D. 


O.F.D. 


S.& 


L. 


Industry 




N % 


N ' % 


N % 


N % 


N 7. 


0 


0 


■ 0.0 


0 


0.0 


0 


0.0' 


0 


0.0 


0 


0.0 


1 ■ 


9 


24.3 


3 


50.0 


2 


13.3 


3 


75.0 


1 


8.3 


2 


2 


5.4 


1 


16.7 


1 


6.7 


0 


0.0 


0 


0.0 


3 


o 

J 


O T 

8 . 1 


1 


Id . 7 


1 


C "7 

6.7 


1 


Id . 0 


0 


U..0 


4- 


4 


10.8 


1 


16.7 


2 


13.3 


0 


0.0 


1 


8.3 


5 . ... 


2 


2.7 


0 


0.0 


2 


6.7 


>o 


0.0 


0 


0;0 


6 


3 


2.7 


0 


0.0 


2 


0.0 


0 


0.0 


1 


8.3" 


7 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


8 


1 


2.7 


0 


0.0 


1 


6.7 


0 


0.0 


0 


0.0 


9 


2^ 


5.4 


0 


0.0 


0 


6.0 


0 


0.0 


2 


16.7 


more th^n 9 


17 


37.8 


' 0 


0.0 ' 


10 


46.7' 


0 


0.0 


7 


58.3 


Is this program accredited by s6me 


board or agency, etc. 












Total 


D.O.D. 


O.F.D. 


. S.& 




Industry' 


¥ 


N % 


N % 


N % 


N 


% \ 




Yes 


56 


39.7 


19 


46.3 


6 


18.2 


21 


—J —J o 

77.8 


10 


25.0 


No 


80 


56.7 


19 


46.3 


27 


81.8 


6 


22.2 


28 


79.0 


Don't know 


5, 


3.5 


3 


7.3 


0 




0 


0.0 


2 ' 


5.0 


r 

Is there any 


V 

fee charged .the 


trainee for taking 


this program? 










Total 


D.O.D. 


O.F.D. 




L. 


Industry 




N . % 


N ■ % 


N % , 


N % 


' N ^ %.. 


Yes 


42 


29.8 


2 ' 


4.8 


14 ' 


42.4 


19 


73.1 


7 . 


■ 17.5 


No 


99 


70.2 


40 


95.2, 


• 19 


57.6 


7 


26.9 


33 


82.5 



ERIC 



44 



36 



TABLE 1 (continued) 



Does the program have a written statement of goals? 





■Total- 


D.O.D. 


O.F.D. 


S. 


& L. 


z I 

Industry 


N ■ % 


N % 


N 


^» 


N % 


N % 


r • 

Yes 


134 


94.4 


42 


100.0 


32 


97 . 0 


27 


100.0 


33 


82.5 


No 


8 


* 5.6 


0 


0.0 


1 


3.0 


0 


0.0 


7 


17.5 


^ ' Don ' t know 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 



QA8 , Are these goals written in behavioral terms, i.e., indicating desired ^hange in 
student/ trainee behavior? 





Total 


D.O.D. 


0. 


F.D. 


S.& 


L. 


Industry 


N* • . % 


N ' % 


N % 


N 7o. 


N 


% 


Yes 


108 


- 76.6 


36 


85.7 


. 29' 


90.6 


22 


81.5 


21 


52.5 


No 


'32 


22.7 


6 


14.3 




9.4 


5- 


18.5 


■ 18 


45.0 


Don' t know 


1 


0.7 


0 


0.0 


0 


" 0.0 


0 


0.0 


1 


2.5 



Based on total N and not on the number who replied Yes 'to previous question. 
QA9 . Are there any formal, prerequisites before a student/ trainee can enter. this program? 





Total 


D. 


O.D. 


O.F. 




S.& 


L. 


industry 


N % 


: N % 


- N % 


N ■ 


% 


N % 


Yes 


85 


60.0 


31' 


73.8 


11 


33.3 


;22 


81,5 


21 


52.5 


No 


57 


40.0 


11 


2^6.2 


22 


66.6 


5 


18.5 


19 


47.5 


Don ' t know 


0 


0.0 


• 0 


0.0 


0 


0.0 


0 


0.0. 


" 0 


0.0 
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TABLE 1 (continued) 



QAIO , • What are they? (prerequisites) 

! 





Total 


D.O.D. 


0. 


F.D. 


S.& 


L. 


Industry 




' N % 


. N %., 


^ N % 


N % , 


N ' % 


At least 8th grade 

w U d W «•>. V./ 1.1 


5 


5.9 


3 - 


9.7 


0 


0.0 


0 


0.0 


2 


9.5 


Some high school 


6 


7.1 


3 


9.7 


1 


9.1 


1 


4 '.5 


1 


4.8 


High school educa- 
tion 


18 


21.2 


2 


6.5 


1 


9.1 


12 


54.5 


3 


14.3 


Some college 


3 


3.5 


0 


0.0 


... 1 


9.1 


1 


4.5 


1 


4.8 


College degree " 


5 


5.9 


0 


0.0 


2 


18.2 


1 


4.5 


2 


9.5' 


Other (specify) 


48 


,, 56.5 


23 


74.2 


6 


54.5 


7 


31.8 


12 


57.1 



QAll , How long does it normally take a student/trainee to complete this program? 





Total 


D.O.D. 


O.F. 


D. 


S.5. 




Industry " 


N 1 


N % 


N % 


N ' % 


• N ^ % 


less than 1 month 


56 


'40.3' 


4 


9.5 


28 


84.8 


0 


o.p 


24 


63.2 


1-3 months 


36 


25.9 


24 


57.1 


1 


3.0 


2 


1.1 


9 


23.. 7 


4-6 months 


,15 


10.8 


11 


26.2" 


2 


6.1 


0 


0.0 


2 


5.3 


7-12 months 


13 


9.4 


,• 3 


7.1 


'"■1 


3.0 


8 


30.8 


1 


2.6 


more than 12 mos. 


19 


13.7 


0 


0.0 


1 ~^ 


3.0 


16 . 


61,5 


• 2 


5.3 1 


don't know 


0 


0.0 


0 


0.0 


0' 


0.0 


0 


0.0 


. 0 


0.0 



4G 
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TABLE 1 (continued) 



About how many hours of in-classroom work are noinnally required to complete 
the program? 





Total 


D 


.O.D.' 


0. 


F.D. 


S.& 


L. 


Industry 




.-N 7. 


•N % 


N ' % 


N 7. 

> 


■ N . % 


less than 10 hours 


14 


10.0 


■ 1 


2.4 


3 


9.1 


0 


0.0 


10 


26.3 


11-19 hours 


11 


7.9 


2 


4.8 


2 


6.1 


0 


0.0 


7 


18.4" 


20-29 hours 


6 


4.3 


1 


2.4- 


3 


9.1 


0 


0.0 


2 


5.3 


30-39 hours 


18 


12.9 


1 


■ 2.4 


12 


36.4 


1 


3.7 


■ 4 


10.5 


40-79 hours 


13 


9.3 


0 


0.0 


8 


24.2 


.0 


0.0 


5 


13.2 


80-119 hours 


10 


7-.1 


3 


' 7.1 


3 


9.1 


1 


3.7 


3 


7.9 


120-159 hours 


6 


4.3 


2_ 


4.8 


1 


3.0 


1 


3.7 


2 


5.3 


160+ 


60 


42.9 


31 


73.8 


1 


3.0 


23 


85.2 


5 


' 13.2 


Don't know 


2 


1.4 


1 


2. "4 


Ov 


0.0 


1 


3.7 


0 


0.0 

■ , 



About how many hours of outside qlassroom, work (e.g., home assignments, reading, 
on-the-job training) are normally required to complete, the program? 





Total 


D.O.D. 


• 0 


.F.D. 


S.& 


L. 


Industry 


N % - 


N • % 


N % 


'n ■ % 


■N , 7. 


less than 10 hours 


44' 


31.9 


6 


14.3 


20 


' 60.6 


. 4 


15.4 


14 


3'7.8 


11-19 hours 


13 


9.4 


1 


2.4 


■ 6 


18.2 


0 


0.0 


6 


16.2 


.20-29 hours 


8 


5.8 


4 


. 9.5 


2 


6.1 


0 


0.0 


2 


5.4 


30-39 hours 


5 


3.6 


2 


4.8 


1 


3.0 


1 


3.8 


1, 


2.7 


40-79 hours • 


11 


8.0 


7 


16.7 


1 


3.0 


1 


3.3 


2 


5.4 


80-119 hours 


10 


7.2 


4 


9.5 


1 


3.0 


0 


0.0 


■ 5 


13.5 


120-159 hours 


11 


8.0 


8 


19.0 


0 


0.0 


2 


7.7 


1 


• 2.7 


160+ 


33 


23.9' 


10 


23.8 


2 


6.1 


17 • 


65.4 


4 


10.8 


Don ' t know 


3 


' 2.2 


vO 


0.0 


0 


.0.0 • 


1 


3.8 


2 


5.4 



47 

39 
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, ^ TABLE 1 (continued) 

1jAl4. 'How would you characterize the instruction in this program? (Circle as many 
as apply) 





Total 


D 


.O.D. 


O.F.D. 


S. 


6c L. 


Industry 




N y 7. 


N % 


N ' % 


N % ■ 


N 7. 


Formal classroom 
teaching 


125 


9^;.0 


39. 


100.0 


30 


.96.8 


27 


100.0 


29 


80.6' 


Programmed instruc- 
tion 


63 


55.3 


- 23 


63.9 


12 


50.0 


16 


72.7 


12 


37.5 

V 


Multimedia usage 






















(e.*g., tapes, 
etc.) 


J7 / 




28 


80 0 


20 


87. 0 


25 


1 nn n 


24 


70.6 


Small group dis- 
cussion 


89 


74.2 


■ 22 


62. 9 


21 


84.0 


21 


84.0 


25 


* 71.4 


Individualized! 
instruction 


11 


70". (T 


24 


66 » 7 


12 


70.6 


24 


96.0" 


17 


' 53.1 


Practical lab/ 






















workshop ex- 
perience 


111 


85.4 


39 


95.1 


20 


76.9 


27 


100.0 


25 


60.4 


On-the-jot' super- 
vised practice 


52 


50.0 


2'4 


66.7 


4 


22.2 


15 


75.0 




30.0 


Other techniques 


29 


55.8 


11 


57.-9 


3 


' 50.0 


. 7 ' 


77.8 


8 


44.4 



*Percentages si^ to more than^ 100— respondents checked as many options as applied. 



-I 
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TABLE 1 (continued) 



9AI5 , Which- of these is the most important element in the program? 





Total 


D.O.D. 


O.F. 


D. 


S. 


& L. 


Industry 


4. 


N % 


N 


% 


N % 


N % 


N % 


Formal classroom 
teaching 




00 0 


9 


21.4 




JO . 1 


/; 


ZZ • z 


r5 


0/ . J 


Programmed instruc- 
tion 


3 


2.1 


1 


2.4 


0 


■ 0. 0 


0 


0.0 


2 


C A 

D.V 


Ml il f^fmprliA n^APP 

i. lU -1- l_ X ill C- CL UOdgC 






















(e.g. , tapes, 
etc.) 


7 


5.0 


1 


2.4 


1 


3.1 


1 


3.7 


■4 


10.0 


Small gropu dis- 
cussion 


5 


3.5 


1 


2.4 


1 


3.1 


1 


3. 7 


2 


5.0 


Individualized 
instruction . 


7 


5i<0 


0 


0.0 


1 


3.1 


3 


11-. 1 ' 


3 


7.5 


Practical lab/ 






















workshop ex- 
perience 


44 


.31.2- 


21 


50.0 


3 


9.4 


15 


55.6 


5 


12.5 • 


On-the-job super- ^ 
vised practice. .* 


10 


7.1 


4 


9.5 


1 


3.1 


1 


3.7 


4 


10,0 


Other techniques 


11 


7.8 


1 


2.4 


5 


15.6 


0 


0.0 


5 


12,5 


Combination of the 
above 


7 


5.0 


4 


9.5 


3 


9.4 


0 


0.0 


0 


0.0 




a the 


program evaluated formally at; 


the end 


of the progra 


m? 




Total 


D.O 


.D. 


O.F.D. 


S 


.& L. 


Indus-try 




N % 


N % 


N % 


N . 


% 


■ N , 


Yes 


110 


78.0 


■ . 39 


92.9 


21 


65.6 


25 


92.6 


25' 


62.5 


No 


29 


20.6 


3 


7.1 


11 


34.4 


2 


7.4 


13 


32:5 


Don' t know 


2 


1.6' 


0 


0.0 


0 


0.0. 


0 


. 0.0 


2 


5.0 



ERIC 
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TABLE 1 , (continued) <' 



Q AI7 , How is the student evaluation carried out? (Circle a.s 'many as apply.) 



i 


Total 


D.O 


.D. 


O.F. 


D. 


S.& 


L. 


Industry 


.1 


N •% 


N ' 


7. 


N ' i 


N % 


N 


7. 


Paper and pencil test 
developed by 
instructor 


65 


45.8 


25 


59,5 


' 3 


9.1 


• 

21 


77.8 


16 


40.0 


Paper and pencil test 
developed by someone 
other than instructor 


41 


28.9 


"l8 


42.9 


6 


18.2 


■^7 


25.9 


10 


• 

25.0 


Overall subjective 
judgment 'by ins"tructor 


53 


37.3 


13 


31.0 


10 


30.3 


14 - 


51.9 


16 


40.0 


Instructor ratings of 
performance, and/or 
products made during 
course 


* 66 


46.5 


21 


50.0 


13 


39.4 


19 


70.4' 


13 


32.5 ' 


Performance (prac- 
tical proficiency) 
examination 


77 


54.2 


33 


78'. 6 


9 . 


27.3 


21 


77.8 


14 


35.0 


Simulation tech- 
niques 


31 


21.8 


■ 6 


14.3 


' 7 


21.2 


11. 


40.7 


7 


17.5 


Oral examination 


19 


13.4 


■ 8 


19.0 


0 


0.0 


5 


18. 5 


6 


15.0 


Other (please 
specify) 


21 


14.8 


4 


- i 

9.5 


9 


27.3 


4 


14.8 


■4 ■ 


;o.o 



Percentages sum to more than 100 — respondents checked as many options as applied, 
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TABLE 1 (continued) 



. Which of these (1-8 above) is the most important element in the evaluation? 



er and pei\cil test 
'eloped by 
t'ructor 

er and -pencil ^test 
developed by someone 
other than instructpx 

Overall subjective \ 
dgment by instructor 

LStructor ratings- of 
irfoinnance, and/or 
roducts made during 
urse 

rformance '(prac- 
cal * prof iciency) 
examination 

Simulation tech- 
Lques 

ral examination 

ther (please 
pecify) 

response 



Total 


D.O.D. 


0, 


F.D. 


S.& 


L. 


Industry 


N - % 


N % 


N % 


N % 


N % 


16 


14.8 


8 

P 


,720.0 


0 


o.a 


4 


16.7 


4 


19.0 ' 


13 


12.0 


3 


7.5 


3 


13.0 


0 


0.0 


7 


33.3, 


. 5 


4.6 


. 0 


0.0 


2 


■ 8.7 


1 


4.2 


2 


9.5 " 


17\ 


15.7 


4 


10.0 


1 


4.3 


1 


29.2 


5 


23.8 


40 

4 . 
0 


37,0 

'\ 

3.7\ 
0.0 


22 
0 

\ 0.- 


55.0 

0.0 
0. U 


6 

4 

. 0 


26.1 

17.4 
0.0 


9 

0 

0. 

'3 


37.5 

0.0 
0.0 


3 

0 
0 


14.3 

0.0 
0.0 


13 
34 


12.0 


3.^ 
2 ~ 


7.5 „ 


7 
10 


30.0 


'3 
3 


12.5 


0 
19 


0.0 
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TABLE 2 



N and Percentage of Training Pr9grams which are Formally Evaluated: 

1) Solely for* ImprovemenL, 2) Solely for Impact, 3) For Both • Improvement 

and Impact, 4) For Neither Improvement nor Impact 



QB(1 and 2)_ 



1) Has ther^ been or' is there currently any formal evaluative effort 
*to improve the program? 

•2) Has there been or is there currently any formal evaluative effort 
to., assess the impact (effects)- of the progx^ml 



Improvement but not 
impact 

Impact but not 
improvement 

Both improviement 
and impact 

Neither improvement 
nor impact 



Total 


D. 






D. 


S.S 


L. 


Industry 


.N "% 


N 




N % 


N % 


N % 


- 26 * 18.3 


' 6 


14.3 


12 


36.4 


. 0 


0.0 


8 


20. 0' 


3 2.1 


0 


o.o'' 


2 


6.1 


0 


0.0 


1 


2.5 

V 


69 48.'6 


24, 


57.1 


11 


33.3 


19 


70.4 


,15 


37.5 


44 " 31.0 


12 


28.6. 


8 


24.2 


■ 8 


29.6 


16.. 


40.0 



By formal, we mean some conscious, planned_e^f f ort though it need not be a sophisticated 
effort. In contrast, an informal evaluation would mainly involve an after-the-fact 
looking over the program and data related to the. program where this activity was not 
part of the overall plan. , 
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TABLE 3 



Description of the Training Program Formal Evaluations for the 
Total Group and Separately for Department' of Defense (D.O.D.)> 
Other Federal Departments (O.F.D. )> State and Local Governments 
and Agencies (S. & L.), and Industry Subgroups. 



QB3 « Who called for this program evaluation to be done? 



Program administrator 

Program's educational 
staff 

Outside- agency 
Other^ 



Program adminis.trator 

Program s educational 
staff 

Outside agency 
OthTar 



I M P R-O -V-E M E -N T 


Total 


D.O.D. 

N=30' 


O.F.D. 

N=25 


S.5. L. 

N=19 


Industry 
• N=24- 


V " 7. 


N % 


N % 


N c % • 


N % 


55 56.1 

45 45:9 

11 ,,:»*#11 . 2 

10^ 20.4 


16 53.3 

7 23.3 
6 20.0 
11. 36.7 ^ 


14 56.0 

17 68.0 

2 8.0 

3 12.0 

3s 


6 31.6 

11 57.9 
3 15.8 
5 • 26.3 


79 79.2 

10 41.7 
0 0.0 

'1 4.2 ■ 








, ' I M P.A C T , 




Total 


D.O.D. . 

' • N=30 


O.F.D. 

N=25 


S,& L. 

N=19 


Industry 
■ N='24 


* 

N 7. 


N % 


. N 7. 


N % 




41 41.8 

/ 24 24.5 '' 
8 8.2 
19 19.4 


9 30.0 

7 23.3 
- "'3 10'. 0 
•11' • 36.7\ 


13 52. Q , 

5 20.0 

1 4^0 
^ 9-0 


6 31.6 

7 36.8 
4 21.1 
6 31.6 


13 54.2 

•5 20.8 
b 0.0 
■2 8.3 



*rN's are infla-^ed slightly and percentages sum to more than 100% because some 
respondents checked raore than one response. Percentages are based on number of 
respondents to this question ^N=98). ^ - . • ^ . 
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TABLE 3 (cont'inued) 



QB4 « Who carried out the evaluation? 









I M P 0 V E M\E,N t' ' 


• 








Total 




d.o.'d. 


■ • .nO'.F.D. 


s. 


& L 


Industry 




- N 


% 


N 7, 


.N 




"n 




N 7, 


Program administrator 


33 


33". 7 


8 


26.7 


10 


40.0 


5 ■ 


26.3 


10 


41. 


7 


Program developers 


35 


35.7 


12 


40.0 


• 


24.0 


, \3 


15. '8 


14 


58. 


3 


, Program's educational 
■ staff 


56 


57.1 


20 


■ 66.7 


17 


68. Q 


9 


47.4 


10 


4- 


1 


Outside agency 


13 


13.3 


8 


26.7 


2 


8.0 


3 


15.8 


0 


10. 


5 


Other . ; 


16 


16.3 


6 


20.0 


0 ■ 


0.0 ' 


' 6 


31.6 


4- 


f • 


6 















IMP A 


C T . • 














1 


1 

Total 


D.O.D. . 


p.F.D. 


S. 


& L. , 


Industry 


i 


N % 


N % 


N 7. - 


N 7. 


N ^ 7. 


Program admiriistrator 


38 


38.8 


9 


30.0 




48.0 


6 


31.6 


11 


45.8 


Program developers 


14 


14.3 


5 


16.7 




8.0 


3 


15. ;8 


'4 


16.7 


Progr'am's educational 
staff 


27 


27.6 


10 


"33.3 

t 




16.0 


8 


42.1 


5 


20.8 


Outside agency 


12 


12.2 


9 


30.0 


1 


•4.0 


-.2 


10.5 


0 


0.0 


Other . , 


18 ■ 


18.4... 


8 


26.7 


0 


OiO 


6 


31.6 


4 


16.7 



4§ 
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TABLE 3 (continued) 



iS5» Who funded the evaluation? 



:interaal funding (part 
b£ program) 

^ernil . funding (out- 
graatj con t "-act, 

) 



[nt:ernal funding (part 

actarual funding (out- 
ide grarft, cpntract, 

/ 



. X M P R 0 V,E M E N T 


-^otal 
' N=93 


D.O'.D. 

N=29 


• ' O.F.D. • 

N='22 


• S.& L. 

N=19, 


Industry, 
N=23 


N % 


N--^ • % • 


N _ % 


N 




',81 87-1 
12' 12.9 


24 ••82.8 
5 17-. 2 


21 95I5 


■18 94.7. 
.1 5.3" 


•18 78.3 
-5 ■■ 21.7 ' 


' \. ■., 
\ . ■ ' . • 

1 .. , 


'IMPACT 


Total 


D.O.D. ■ 

N=23 


I O.F.D. 

• N=13 


S.& L. 

■N=18 


I-ndustry 
N=13 


« 

N % 


N % 


N %. 


N , %. 


N % • 


6C 89.6 

♦if 

7 -10.4 


0 20 87.0 
3 13 . 0 


12 •^ 92.3 
1 7.7 ■ 


■16 88.9 
. .2 11.1 


12 92.3 
1 - 7.7 



/ 



/ 
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TABLE 3 (continued) 



QB6 , ' Was the evaluation monitored by ^some person or group independent of the 
• evaluation? 



Yes 
No 

DonVt know 





' ^IMPROVEMENT 




Total 
N=93 


D.O.D. 

N=3b 


o:f.d. 

■N=22, 


■.S.& L. 
N=19 


! Indus Cry 
/ N=22 


N % ■ 


■N / %■ 


N • 7. 


N % 


I.N % 


39 41.9 
50 53.8 
4 4.3; 


19 ^ /63.3 _ 
10 /33.3 
1 / 3.3 


lO' ^-45.5 
11. 50.0 

■ 1 '^A. 


7 36.8 
10 52.6 
2 10.5. 


i 3 . 13.6 
19 . 86.4 
0 0.0 



Yes 
* No 
Don ' t know 



Total 
N=73 



N 



29 \39j7; 
41 B6i2| 
3^ 



IM PACT 



D=0.D. 

N=25 



N 



% 



13 52.0 
11 44.0 
1 4.0. 



O.F.'D. 

N=13 



N 



3 
9 
1 



23.1 
69.2 
7.7 



S.& L. 

N=19 



N 



% 



9 47.4 
9 47.4 
1 5.3 



Industry 
N=16 



N 



4 25.0 
12 75:0 
0 0.0 



\ 
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TABLE 3 (continued) 

■ A 

QB7 . What year(s) did the evaluation occur? 





Total • 


D. 


O.D. 


O.F.D. 


S 


& L. 




Industry 


> 


N 7. 


N % 




% 


N 7. ■ 


'N 


7, 


Before 1965 


1 


1. 


1 


0 


0.0 


0 


0.0 


0 


0. 


0 


1 

X 


4. 


3 


1965 


o-ii 


0. 


0 


0 


0.0 


U 


U . 0 


0 


,0. 


0 


0 


0. 


0 


1966 




1. 


1 


0 


0.0 


U 


U , U 


0 


0. 


0 


1 


4. 


3 


1967 


t 

0 


0. 


0 


0 


0.0 


U 


U . 0 


0 


0. 


0 


0 


0. 


0 


1968 


0 


0. 


0 


0 


0.0 


n 
\j 


n n 


0 ■ 


0. 


.0 


n 
u 


0. 


0 


1969 


0 


0. 


0^ 


0 


0.0 


0 


0.0 


0 


0. 


0 


0 


0. 


0 


























1970 


0 


0. 


0 


0 


0.0 


0 ' 


0.0 


0 


0. 


0 


- 0 


0. 


0 


1971 


4 


. 4. 


2 


1 


3.3 


1 


4.3 


■0 


0. 


q 


2 


8. 


7 


1972 


9 


9. 


5 


2 


6.7 


1 


4.5 


1 


^5. 


3 


5 


21- 


7 


1973 o 


14 


14. 


7 


6 


20.0 


5 , 


21.7 ■ 


2 


lo. 


5 


1 


4. 


3 


1974 


66 


69. 


5 


21 


70.0 


'16- 


69.6 


16 


84. 


2 


13 


56. 


5 



Most recent response recorded. 



QB8. iiow long did" it take? 







Total \ 
N=74 




.O.D. 

M=28 ■ 


0 


.F.D. 

N=16 


■ S 


.& L. 

N=16 , 


Industry 
N-13 






N 


. 7' 


! N 


7, 


N 




N 


7. 


N ■ % 


Less than 1 


week 


7 


9.5 


0 


0.0 


1 


6.3 


3 

\ 

\ 

\ 


"^18.8 


■3' 


21.4 


less than 1 
than a week) 


month (more 


4 


5.4 


1 


3.6 


. 1 


6.3 


6.3 


1 ■ 


7.1 , 


1+ months 
2+ months 




4 
3 


5.4 
4.1 


■ 0 
2 


0.0 
7-1 


2 
0 


12.5* 
0.0 , 


1 


\l2.5 

■\ 


' 0 
' 0 


0.0, 
0.0 


3+ months 




1 


1.4 


0 


0.0 


0 


0.0 


0 


^ °\ 


^ 1' 


7.1 


4+ months 




1 


1.4 


0 


0.0 


0. 


■..0 . 0 


0_ 


^ 0.0 


\sl- 


7.1 


5+ months 




11 


14.9 


' 3 


■ 10.7 


3 


18.8 


1 


6.3 


4X 


.28.6 


6-12 months 




7 


9.5 


4 


14.3 


0 


■ 0.0 


' 2 


12.5 


1 




More than 1 


year 


5 


. 6.8 


3- 


10.7 


0 


0.0 


2 


12,5 


0 


0.0^ 


A continual 
process 


■k 

or ongoing 


31 


41.9 


15 


53.6 


9 


56.3 


4 


25.0 


3 


21.4 



'^Ongoing" was sometimes specifically written in the blank. 



TABLE 3 (continued) 



QB9. Is it still continuing? 



I>1PR0VEMENT 












Total 
N=95 


D.O.D. 

N=30 


O.F.D. 

N=23 


S.& L. 

N=19 


Industry 
•N=23 ■■ 


N % 


N % 


. N % 


N % 


N • % 


81 85 . 3 
14 14.7 


26 86.7 
4 -13.3 


22 95.7 
1 4.3 


14 73.7' 
5 26.3 


19' 81 . 3 
4 18.7 





IMPACT^^ 










Total, 
N=71 


D.O.D. 

■ N=24 ' 


O.F.D. 

.■N=73 


S.&' L. 
N=18 


Industry 
N=16 


N % 


N % 


N % 


N- % 


N % 


57 80.3 
14 19.7 


20 83.3 
4 16.7 


10 76.9 
3 23.1 


14 . 77.8 
4 2 .2 . 2 


■ 13 81.3 
3 18.7 
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TABLE 3 (continued) 
QB10 « ''Who or what the focus of measurement? 













I M P R 0 V. 


E M E N T 












Total 


D.O.D. 


p 


.F.D. 


S. 


& L. 


Industry 


N % 


N % 


N % 


N ,% 


N % 


Students 


62 


63.3 


19 


63.3 


16 


64.0 


7 


36.8 


20 


' 83.3 


Classroom & teaching 






















processes 


45 


45.9 


12 


40.0 


11 


44.0 


.13 


68.4 


9 


37.5 


Teabhers 


33 


33.7 


10. 


33.3 


11 


44.0 


8 


42.1 


4 


16.7 


Curriculum 


57 


58.2 


22 


73.3 


13 


52.0 


15 


78.9 


7 


29.2 


Other 


13 


13.3 


6 


20.0 


0 


0.0 


6 


31.6 


1 


4.2 



Students 

Classroom & teaching 
processes 

Teachers 

Curriculum 

Other 





IMPACT 




Total 


D.O.D. 


O.F.D. 


S.& L. , 


Industry 


■ N % 


N ■ % 


N % 


N % 


N . % 


54 55.1 

19 19.4 
16 10.3 
30 30.6 
18 18.4 


20 66.7 

4 i3.3 

6 20.0 
11 36.7 

7 23.3 


7 28.0 

4 10. 0' 

2 8.0 

3 12.0 

5 20. 0 


12 .63.2 

8 42.1 
. 5 26.3 
10 52.6 

5 26.3 


15 62.5 

3 • 12.5 
3 12.5 
6 25.0 
1 ,4.2 



"7 ■ 
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TABLE 3 (cantinued). 



Q Bll * What measurement . techniques were used? 



1= 



"Teacher made'* paper and 
pencil tests. 

Standardized paper and 
pencil ttjsts 

Interviews and 
ques tionnair es 

Observations 

'Rating scaler's 

Other 



•IMPROVEMENT 



Total 


' D 


.CD. 


O.F.D. 


S 


& L. 


Industry 


N % 


N ' % 


N % 


N % 


N % 


10 10. 


2 


2 


• 6.7 


2 8.0 


. 4 


21.1 


2 ' 


8.3 


20 20. 


4 


7 


23.3 


6 24.0 


. 2 


10.5 


5 


20.8* 


63 64. 


3 


'21 


70.0 


12- 48.0 


10 


52.6 


20 


83.3 


59 60. 


2 


16 


53.3 


15 ' -60.0 


16 


84.2 


12 


50.0 


22 22. 


4^ 


- 6 


20.0 


4 16.0 


,6 


31.6 


6 


25.0 


15, 15. 


3 


4 


6.-7 


6 24.0 


■ 4 


21.1- 


1 


4.2 















I M P A 


C T 
















To.tal 


D-.O 


.D. 


0 


.F..D. 


S. 


& L. 


Industry 




N % 


N . % 


N ■ % 


N ■ % 


N % 


"Teacher made" paper and 
pencil tests 


3 


3.1 


1 - 


3.3 


0 


0.0 


2 


10.5 


0 


0.0 


Standardized paper and 
pencil tests 


= 8 .. 


. 8.2 


4 


13.3 


0 


0.0 


0 


0.0 


4 


16.7 


Interviews and 
questionnaires 


48 


49.0 


17 


56.7 


7 


28.0 


Id 


52.6 


14 


58.3 


Observations 


35 


35.7 


8 


26.7 


4 


16.0 


12 


63.2 


11 


45.8 


Rating stales 


21 


21.4 


6 


20.0 


3 


12.0 


5-, 


26.3 


7 


29.2 


Other 


.13 


13.3 


2 


6.7. 


6 


24.0 


3 


15.8 


i. 


8.3 
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TABLE 3' (continued) 
QB12 , Was the reliability of the measures assessed? 



Yes 
No 

Don't know 



No 

Don^t know 





i-LM P R 0 V E M E N T 








Total 
N=93 


, D.O.D. 

N='29 


O.F.D.', 

N^23 ■ ■ 


S.& L. . 

N=15 ■ 


•' Industry 
N=22 


N % 


N % ' 


N Z 


N % 


N % 


38 40.9 
43' 46.2 
12 . 12.9 


14 .58.6- 
8 . 27.6 
4 13.8 


1 30.4 
16 69.6 
0 0.0 


7 36.8 
9 47.4 
3 15.8 


i 31,8 
10 45.5 
5 31.3 


c. 




IMP-ACT 


Total 
N=70 


D.O.D. 

N=24 


■O.F.p. 

N=12 


S.& L. 

N=18 


Industry 
N=16 


N ■ . % 


N % 


N , , 7. 


N 7. 


- N._ % 


32 45.7 
23 32.9 
.15 21.4 


13 54.2 
7 29.2 
. 4 16.7 


5 41.7 
5 41-. 7 
2 16.7 


6 33.3 
•8 44.4 
4 22.2 


8 50.0 
3. 18,8 
5 31.3 



ERIC 
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TABLE 3 (continued) 
QB13 « Was the validity of the measures -assessed? 



Yes 
No 

Don^t know 



Yes 
No 

Don * t know 





. I M -P R 0 V ' E M E N T 


' Total.-" 
N=93 '. 


D.O.D. 

N=29 


O.F.D. 

N=23 


• S.& L. 

N=19- 


Industry 
N=22 


N % 


^ N % 


N % 


N % 


N 7. 


4 44.1 
39 . 41.9 
13 14.0 


20 69.0 
5 17.2 
4 13.8' 


6 26.1 
16 69.6 
1 4.3 


,9 67. ,4 
8 42.1 
2 10.5 


6' 27:3 
1 10 45.5 
6 -27". 3 






' • I M P .A C T 








Total 
_ N=70' 


D.O.D. 

N=24 


O.F.D; 

N=12 


S.& L. 

N=18 


Industry 
N-16 


N % 


" N j: - 


N ' % 


N % 


N % 


33 ■ 47.1 
21 30: 0 
16 22.9 


15 62.5 
5 20.8 
4 16. 7 


5 41.7 
4 33.3 
3 25.0- 


- 7 38.9 
8 ■ 44.4 
3 . 16.7 


.6 ',37.5 
4 ''25.0 
'6 37.5 



ERIC 
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table; 3 Ocontinued) 
QB14 . Did the"' measure's assess the program's goals? 



Yes 

No. . 

Ik ' 

Don't know 



IMPROVEMENT 



Yes 
No 

Don * t know 



Total ■ 
° N=94 


D.O.D. 

N=29 


O.F.D. 

N=23 


S.& L. 

N=19 


Industry 
N=23 




N 7, 


N % 


N % 


N % 


75 79.8 
14 14.9 
5 5.3 


25 86.2 
3 10.3 
1 3.4 


15 65 ..2 
7 30.4 
1 ' 4.3 


17 89.5 
0 0.0 
2 10.5 


18 ■ 78.3 
4 1.7.4 
1- 4.3 






IMPACT- 








Total 
N=72 ■ 


D.O.D. 

N=24 


' O.F.D. 

.. N=13 


S.5. L. 

' ' N=18 


Industry 
N=17 


N % 


N % 


N 7. 


N . % • 


. N 7. 


62. 86.1 
3 4.2 
.7 .9.7 


21, 87.5 
■ 2... 8.3 
1 4.2 


11 84.6 
0 0.0 
2 15.4 


15 83.3 
0 0.0 

3 ., 16.7 


15 da. 2 

1 5.9 
1 5.9 



ERIC 
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^ - , TABLE 3 (continued) 

QB15 , Did the measures assess potential side effects? 



Yes - 
No 

Don' t know 



Yes 
Na 

Don't know 





IMPROVEMENT 


Total, 
N=92 


D.O.D. 

N=29 


'O.F.D. 

N=23 


S.& L. • 

N=18 


Industry 
N=22 


N ' ■ 7. 


N % ; 


N t 


N ^ % 


J N %' 


36. 39.1 
39 42.4 
17 18.5 


13 44.8 
10 '34.5 
6 20.7 


6 26.1 
16 69.6 
1 4.3 


10 55.6 
3 16.7 
■5 27.8 


7 ' 31.8 
" 10 45.5 
5 22.7 


■ . ^ ■ . . ■ . 

t 




IMPACT 








Total 
' N=71 


D.O'.D. 

N=24 ■ 


-O.F.D. 

N=13 


S.& L. 

N=18 


Industry 
N=1'6 


N „ ' 


N , % " 


N % 


N ' % 




36 50.7 
_ 17 ' 23'. 9 
18' 25.4 


12 50.0 
5 20.8 
7 29.2 


8 61.5 
3 23.1 
2 15.4 


'9 50.0 
3 16.7 
6 33.3 


1 43. '8 
6 37.5 
3 18.8 



56 
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TABLE 3 (continued) 



QBIS , What research design wds used in- this evaluation? 



True experiment 
Qua s t- exp er imen t 
Observational study 



True experiment 
Quasi-experiment 
Observational study 



IMPROVEMENT 


Total 


D.O.Di 


O.F.D. 


S.5. L. 


Industry 


N % 


,N % 


N % 


N % 


N • % 


14 15.9 
11 12.5 
63 71.6 


9 ^27.3 
6 18.2 
23 54.6' 


1 6.3' 
0 0.0 
15 ' 93.7 


3 13.6 
'2 9.1 
17 77.3 


1 5.9 \ 
3 17.7, 
13 ' 76.4 


t 

- . '. ' . 


^ ■■IMPACT 


Total 


D.O.D. 


O.F.D. 


S.6. L. 


.Industry 


N % 


N % 


\ N % 


; N % 


N ■ % 


10. 15.4 
9 13.9 
4-6 70.7 


5 22.7 
4 18.2 
13 59.1 


0 0.0 

1 10.0 
9 90.0 


3 16". 7 
1 5.5 
14 77.8 


• 2 13.3 
3 20.0 
10 66.7 
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TABLE 3 (continued) 
Was there a statistical analysis of the results? 



\ 

\ 

Yes \ 
No \ 
Don't know 



1 




IMP 


R ,0 V E M E N T 










Total 
N-=91 


D.O.D. 

N=29 


OlF.D. 

^ ,N=23 


S.& L. ■ 

N=18 


Industry 
N=27 


N % 


N > % 1 


N % 


N % 


N i 


24 26.4 
56 61.5 
11 .12.1 


10 34.5 1, 
14 48.3 
,5 17.2 


3 13.0 
, 19 82.6 
1 4.3 


2 11. 1 
14 77.8 
' 2 11.1 


9 " 42.9 
9 42.9 
3 14.3 



Yes 
No 

Don' t know 





IMPACT 








Total 
N=68 


D.O.D. 

-N=24 


O.F.D. 

N=13 


S.& L. 

. N=16 


Industry 
— N=15 


. tT % 


N ■% 


; N % 


N % 


N :% 


2? 32.4 
36- 52.9 
10 14.7 


9 37.5 
9 41.7 
5 20.8 


i 5 38.5 
! 7 53.8 
: 1 . 7.7 


3 18.8 
12 75.0 
1 6.3 


5 33.3 
7 46.7 
■ 3 20.0 



^ 60 
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TABLE 3 (con%fi.nnedi 



QB20 , Was there a written report of the evaluation? 



Yes 
No 

Don't know 





IMPROVEMENT' , 








Total 
N=92' 


D.O.D. 

■ N=28 V 


O.F.D'. 
■• N=23^ 


S.& L. . ■ 
N=18 


Industry 
N=23 


N ^ % ' 


■ N 7. 


N % 


N % 


N ■ 7. 


44 47.8 

45 48.9' 
3 3.3 


16 57.1 
10 35.7 
2 7.1 


9 39^1 
13 56.5 
1 - 4'. 3 


10 55:6 
8 '^44.4 
0 0.0 


9 39.1 
14 60.9 
■ 0 ^0.0 





IMPACT " ■ \ ; 








Total 
N=68 


D.O.D. 

N=22 


O.F.D. 

■■ N=13" ■ 


S.&|L. 
N=l4 


Industry 
N=17 


' N _ 


N % 


% 


N % 


N % 


33 48.5 
31 45.6 
4 5.9 


10 45.5 
10 45,5 
2 9.0 


8 61.5 
4 30.8 
1 7.7 


9 ■ 56^.3 
' 6 37.5 
l' ''6.3 


6. 35.3 
11 64.7 

0 o.d 
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TABLE 3 (continued) 



QB21 , , Could we have a copy? 



Yes 
No 

Don't know 



'IMPROVEMENT' 








Total 
N=49 


D.O.D. 

N=17 


O.F.D. 

: N=10 


S.& L. ^ 

N=12 


Industry 
N=10 


N % 


N % 


' N*^ % 


' N ' % 


V— 

N % 


10 20.4 
30 61.2 
9 18.'4 


• 4 23.5 
'? ■ 41.2 
6 35.3 " 


.2 ' 20.0 
7 70.0 
1 10.0 


2 16;7 
9 75.0 
1 8.3 


. 2 ' ■2b,o 

7 70.0 
1 10. .0 



Yes 
No 

Don't know 





■ .IMPACT 








Total 
- N=3S 


. D.O.D.- 

■ N=12 


O.F.D. , 

N=9 


S.& L. 

; N=ll 


■ Industry 
N=6 


,N % 


N % ^ 


N ^ % 


N 7. 


N '/, 


9 23.7 
;22 57.9 
1 7 ■18.4 


3 25.0 
5 41.7 ^ 

4 33.3 


' 3 33.3 
5 55.6 
1 11.1 

t 


2 18.2 
8 72'. 7 
;1 ■ 9.1 


.1 16.7 
4 /^6.7 
.1 16.7 



A. 
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/ . TABLE 3 (continued) 

Did ■■any changes take place as a result of the evaluation? 



IMPROVEMENT 





Total 


, D.O 


.D. 




O.F.D. 




S. 


& L. 1 


Industry 




N=91 




28 




N= 


23 






=17 


N=2.3 




N % • 


N % 


N % 


N % 


N • % 


Yes y 


74 81.3 


23 


82. 


1 


20 


87. 


0 


15 


8 8. '2 


: 8 ■ 50.0 


No 

* 


9 , 9.9 


4 


14. 


3 


2 


8. 


7 


2 


11.8 


2 12.5 


Don ' t know 


8 ■ 8.8 


1 


3. 


6 


1 


4. 


3 


0 


^ 0.0 


"6 37.5 



Yes • 
No 

Don ' t know 



IMPACT 









Total 
N=66 


D.O.D. 

N=22 


O.F.D. 

N=12 


S.& L. 

N=16 


Industry 
• N=16 


N % 


N % 


■ N X ,. % 


N % 


N % 


40 50,6 
14 21.2 
12 ' 18.2 


15 68.2 
/ 4 18.2 
3 13.6 


5 41 . 7 

6 50.0 
1 8.3 


12 75.0 
2 12.5 
0 0.0 


8 50.0 

2 12.5 ■ 
6 37.5 
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TABLE 4 



.N and Percentage of Training Programs which are Informally 

/ 

Evaluated: 1) Solely for Improvement, 2) Solely for Impact, 

. i • V 

3)^ For Both Improvement and Imoact, 4) For Neither Improvement 

■ / I 

nor Impact / ! 



QB(24 and 28) 



24) Has there been or is there currently any informal evaluatio/n. 
effor:t to Improve the program? 

28) Has there iD'een or is thete currently any informal evaluatzLon 
effort to assess the impact (effects) of the program? / 





Total 




D. 


O.D. 


C 


; 

.F..D. 

/ 


S 


& L. 


Indiiistry 




N 


% . 




N . % 


N 


/ - 


N 


& 


N^' % 


Improvement but not 
impact 


18 


12. 7 




3 


. 7.1 




1 18.2 


3 


11.1 


/ 


15.0 


Impact but not 
improvement 


, 1 


0.7 




0 


0.0 


) 

0 


0.0 


1 


3.7, 


/ 


0.0 


Both improvement 
and impact 


22 


15.5 




9 


21.4 




6.1 


4 


14.8 




17.5 


Neither improvement 
nor impact 


2 


1.4; 


0 


0.0 


ol 


0.0 


0 


0.0 ./ 


/; 


5.0 


Does not apply 


99' 


69 . 7 ' 


30 


71.4. 


25 


\ 75.8 


19 


70.4 1 


25 


62.5 



7,0 

0 



0/ 
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TABLE 5 



Descriptive Information about Technical Program's 
Informal Evaluation (from those who responded that 
there\/ere no formal evaluations) 



QB(25 and 29) > 25) Who carries out this informal improvement effort? 

29) Who carries -out this informal impact evaluation? 



Instructors 
Administrators 
Some third party 





IMPROVEMENT 




Total ■ 
N=40 , 


D.O.D. 

N=12 


O.F.D. . 

N=8 


S.& L. 

N=7 


Industry 
.N=13 


N _% 


N 7. 


N % 


N % 


N % 


35 87.5 
28 70.0 
7 17.5 


10 83.3"^ 
' 9 75.0 
1 . 8.3 


7 87.5 

6 75.0 

% 

0 0.0 


7 100.0 
3 42.9 
2 28.5 


11 84.6 
10 76.9 
4 30.8 



N based on number oS respondents who answered yes to Question 24, 



Instructors 
Administrators 
Some third party 





I P A C T 




Total 
N=23 


D.O.D. 

N=9 


O.F.D. 

N=2 


S.& L. 

N=5 


Industry 
N=7 




AA 

N % 


AA 

N % 


AA 

N % 


AA 

N % 


16 69.6 
16 69.6 
8 34.8 


5 54.4 

6 66.7 
.1 11.1 


2 100.0 
2 100.0 
1 -50.0 > 


5' 100.0 
4 80 . 0 
2 *40.0 


3 42.9 

4 57.1 
4 ■57.1 



AA 



.. N based on number of respondents' who answered yes to Question 28, 



71 
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TABLE 5 (continued) 



\ 

What information is used to guide -improvements? 





Total 


D.O.D. 


. O.F.D. ' 


s. 


& L. 




Industry 






40- 




N=12 




N=8 


N 


=77 




N= 


=13 




N % 


N % 


N % 


N 1 




N % 


Students' judgments 


28 


70.0 


10 


83.3 


6 


75.0, 


4 


57. 


1 


8 


61.5 


Students' scores 


21 


52.5 


7 


58. ,3 


4 


50.0 


2 


28. 


5 


i 8 


61.5 


Instructors' judgments 


37 


92.5 


12 


100.0 


7 


87.5 


7 


100. 


0 


' 11 


84.6 


Judgments of other 
























observers (e.g.. 


25 


62.5 


9 


75.0 


2 


25.0 


5 


71. 


4 


9 


69.2 


administrators) 
























Subsequent reports 
























from supervisors on 


19 


47.5 


6 


50.0 


4 


50.0 


4 


57. 


1 


4 


38.4 


the job 


















V 







N is based on number of respondents vho answered yes to Question 24. 



72 

64 



APPENDIX A 
PROGRAM EVALUATION QUESTIOIINAIRE 

Concerning }jour progra m 

, A Description of the Program 
!• When did this program, as currently offered, first begin? 

2, About how many students/ traineejs have been trained in this program so far? 

a. What percentage of students/trainees complete the program successfully? 

b. Can. unsuccessful trainees repeat the. program? (Circle one) 

Yes 1 

No 2 

Don't know 3 

If so, what percentage of unsuccessful trainees repeat the program? 

"3. Is this program offered in more than one center or location? 

Yes 1 

No 2 

Don't know 3 

If so, in how many other centers or locations is it offered? 

4. Is this program accredited by some board, agency, etc.? 

'■ ' % 
Yes 1 

No . . 2 

Don't know ; .... 3 

- _ 

5^ Which boards or agencies? (please specif y)^_^^ 

6. Is there any fee charged the trainee for taking this program? 

Yes \ .... . 1 

No ....... • 2 

7. Does the program have a written statement of goals? 

Yes. ... I . 1 

No ; ^ . .2 

^ Don't know \ ^ ... 3 
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1-4 

5-7 

8-11 
12-13 

14 

15-16 



17 



18-19 



20 



21-22 



23 



24 



Are these goals written in behavioral terms, i.e., indicating- desired changej 
in student/trainee behavior? 

Yes 1 

No ... 2 

Don ' t know 3 

Are there any formal educational prerequisites before a student/trainee can 
enter this program? t 

YeiT"^. 1 

\ 2 ^ 

Don' t know 3 



WTiat are they? 

At least 8th grade education ...... 1 

Some high school 2 

High school graduation 3 

Some college 4 

Cplle'ge degree .5 

Other (please specify) 6 



How long does it normally take a student/trainee to complete this program? 

Less than 1 month 1 

1-3 months . 2 

4-6 months . 3 

7-12 months. . 4 

More than 12-'months. 5 

Don ' t know 6 - 



A-2 
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About how many hours of in-classroom work are normally required to complete 
the program? 

Note: 40 hours =» 1 week full time ^ 

^ - 80 hours = 2 weeks full-time 

120 hours =^ 3 weeks full time 
1 160 hours = 4 weeks full time 

Less than 10 hours 1 

11-19 hours 2 - 

. . 20-29 hours 3 

30-39 hours . 4 

40-79 hours . .5 

80-119 hours. ..... . 6 

120-159 hours ..... 7 

160+ hours 8 

Don ' t know. . . ' 9 

How many hours of outside classroom work (e.g. / home assignments, reading, 
on-the-job training) are normally req^uired to complete the program? 

Less than 10 hours. ".1 

11-19 hours . \ 2 

20-29 hours , . . . 3 

. 30-39 hours , . . . 4 

40-79 hours .................. 5 

80-119 hours. 6 

120-159 hours . . : 7 

160+ hours 8 

Don't know. '.9 " > 



A-3 
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14, "How would you characterize the instruction in this program? (Check as 



many as apply) 

Formal classroom teaching (1) 

Programmed instruction (2) 
Multi-media usage (e.g. , .tapes, etc.) (3) 

Small group discussion (4) 

Individualized instruction . (5) 

Practical lab/workshop experience (6) 

On the-job supervised practice (7) 

Other techniques (please specify) (8) 



1 

Yes 



2 

No 



Don ' t know 



31-46 



15. Which of these is the most important element in the- program? (Circle one) 



Formal claSvSroom teaching.. . . . . . . 1 

Programmed instruction ....... ^ 2 

Multi-media usage 3 

vSmall group discussion ^ . 4 

Individualized instruction . » ; .. . 5 

Practical lab/workshop experience. . . 6 

On^the'-job supervised practice 7 

Other techniques (please specify). . . 8 



47 



16. Are students in the program evaluated formally at the end of the 
program? 

Yes.. ■ . . . 1 

No 2 

Don ' t know .............. 3 



48 
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How is the student evaluation carried out? (Circle as many as apply.) 



Paper and pencil test developed by instructor 1 

Paper and pencil test developed by someone other than 

instructor 2 

Overall subjective judgment by instructor. ........ 3 

Instructor ratings of performance, and/or products made 

during course ^ 

Performance (practical proficiency) examination.. .... 5 

Simulation techniques 6 

Oral examination • • 7 

Other - (please specify) • 8 



Which of these (1-8' above) is the most important element in the evaluation 



77 



Program Evaluation 

Mow I'd like to ask you some questions concerning the evaluation for this 
itselO' where the purpose of the evaluation is to assess the program 

1, Has there been or- is there currently any formal evaluative effort to 
Improve the program? 

Yes. 1 
No .2 

* 

2. Has there been or is there currently , any formal evaluative effort to 
• assess the impact (effe,cts) of the program? 

Yes .1 

^ Uo\' 2 

If you answered No to both thes^ questions, go on to Q 24 if you answered ' 

Yes to either or both , please con-tinue. Usa Column A in reference to evaluation 
to improve the program and Column B in reference to evaluation to assess the 
program's impact. 



58 
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Column A 
Program 
Improvement 
1 



3. Who called for this evaluation to be done? 



Column B 

Program 

Impact 



Program administrator 
Program's educational staff 
Outside agency 
Other(please specify) 



(1) 

(2) 
(3) 
(4) 



(1) 
(2) 
(3) 
(4) 



60-67 



By formal, we mean some conscious, planne"ci effort though it need not be a 
sophisticated effort'. In contrast, an informal evaliiation would mainly involve 
kn -after-the-fact looking over. the program and data related to the program 
where this activity was not part of the overall olan. 
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Column A 

Program 

Improvement 

1 ^ 



Column B' 

Program 

Impact 



4, .Who carried out the evaluation? 

Program administrator 
Program developers 
Program' s educational staff 
Outside agency 
Other (please specify) 



(1) 
(2) 
(3) 
(4) 
(5) 



(1) 
(2) 
(3) 
(4) 
(5) 



68-77 



5. Who funded the evaluation? 

Internal funding (part of program) 

External funding (outside grant, contract, 
_e.ta. ) 

6, Was the evaluation monitored by .some person or 
group independent of the evaluation? 



Yes 
No 

Don ' t know 



(1) 
(2) 



(1) 
(2) 
(3) 



(1) 
(2) 



(1) 
(2) 
(3) 



78-81 



82-87 



7. What year(s) did the evaluation ■• occur? 



88-92 



how long did it tak.e?_ 



93-94 



Is it still continuing? 



Yes 
No 



(1) 
(2) 



(1) 
(2) 



95-98 
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Concerning Measures and Desig;n 



Who or what was tshe focus of measurement? 
Students 

Classroom anci, teaching processes 

Teachers \^ - 

Curriculum— -V" 
Other (please 3pecify)_ 



Column A- 
Ptogram 
Improvement 
1 



.(2) 
.(3) 

(5) 



Column B 

Program 

Impact. 



.(1) 
.(2)- 

.(3) 

.(4) 

(5); 



99-108 



What measurement techniflues were used? 

"Teacher made" papier and pencil tests 

-S£a-nd-a.^Edized paper^and pencil tests 

Interviews~and questionnaires 
Observations 
Rating scales 
' Other (please specifyV 



(1) 
(2) 
(3) 
(4) 
(5) 
(6) 



(1) 
(2) 
(3) 
(4) 
(5) 
(6) 



109-120 



Was the reliability of the measures assessed? 



Yes 
No 

Doit't know 



Was the validity of the measures assessed? 



Yes 

No , 

Don'-t .know 



(1) 
(2) 
(3) 



(1) 

(2) ' 

(3) ; 



(1) 
(2) 
(3) 



(1) 
(2) 
(3) 



121-1?6 



127 -132 



Did .the measures assess the program's goals? 



Yes 
No 

Don ' t know 



CD 

(3j 



(1)- 

(2) 
(3) 



133-138 
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15. Did the measures assess potential side-effects? 

Yes 

■ ; . No . 

■ \* . Don ' t know 



Column A 
Program 
Improvement 
1 

(1) 

(2) 

(3) 



Column B 

Program 

Impact 



(1) 

(2) 
(3) 



16. What research design was used in the evaluation? 

4- true experiment .Yes, 

No 

Don^t know 
Yes . 
No • 

Don't know 
Yes 
No 

Don ' t know 



b . quasi-exper iment 



c. observational study 



•i 

(1) 

(2) 
(3) 
(4) 
(5) 
(6) 
(7) 
(8) 
(9) 



(1) 
(2) 
(3) 
(A) 
(5) 

(6) 

(7) 
(8) 
(9) 



17. Will you briefly describe the evaluation? 



■18; Was there a statistical analysis of the results? 

■ ' . ■ No- - ' 

Don^t know., 

19. (If Yes to 18) .What statistical techniques were used? 



(2) 
(3) 



(1) 
(2) 
(3) 



81 



A- 9 



Concerning Reports ^ 



/ . Program 

/ ^ . ' ImproveniGnt 

/ • ' • ' • 

'. Was there a written report on the evaluation? 

•'^ ' ' -..Yes •(!) 

. _ s NO (2) 

Don' t know (3) 



21. Could we have a copy? ^^"^^ " (1) 



1. * • / „ ^ 



KSend prepaid envelope if yes.) 



No . ■ (2) 

Don' t know . (3) 



22.- Did any changes take place as a result of the 

evaluation? ^^ 

Yes 

NO 

, - Don't, know (3) 



25. (If yes to 22) What were they? 



F.OR THOSE -WHO ANSWERED NO TO QUESTIONS 1 , AND 2. 
24. Has there beeh or is there currently any inforoial 
• evaluation effort to improve the program?. 

^es ^ 

2 



25. 'who carries out this informal improvement effort? 
/ •;• Instructors 

\ 2 

■ .Administrators . . . » .- • • • • •. • * " , 
3 * Some third party . . 3 
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• Vt^at infonuf tion is used 'to guide the 

■ ■ • Students' judgments.. .... 

: Sttfdeuts* scores • • \ • • • 
, " Instrucdors* judgmen.ts • . • 
Judgments of other, observers 
(e.g. J administratorsX . . * - 
_ Subsequent reports from supervisors 
: , on the job , 



17., Can you cell me anything more about this informal 
effort? ^ ■/ • 



' N 

28. Has t.feere been or is there currently any informal 
evaluation effort* to a^ess the -impact (effects) ,^ 

of . the__pr.ograiat • 

r Yes* ^« • ^ 

No . * . 2 

Don ^ t know ^ 

25* -who carries out tfils informal evaluation? 

Instructors*- 

Administrators 2 
Some third party 3 



30. ■ Caa you tell me anyljhiag moxB about this informal 
evaluatiou effort? } ' 



APPENDIX B 
GENERAL LETTER OF ENQUIRY 



I- 



E DL- C ATI ON AL TKSTIXG ShZnVlC R 



Area Code 609 
92] - 9000 
CABLE^EDl ClEyrSVC 



PHINCE^hrcAX. X.J. 08 



Division 
Educationa I Stu diej 



Dear Director: y 

/ ^ ' ■ ■ / - ^ A: \ - - \ 

As was indicated to ypu in a recent telephone conversation, we are \ 
conducting a survey of training programs as part of a larger jstudy sponsored 
by the Office Naval Research. Specifically, we are trying' to identify \ 
a substantial number o'f recently Introduced technical training programs \ 
.^or adults. \|e understand that your institution sponsors or/ conducts such \ 
-programs, and we would be grateful if you could let us have ,k catalog or 
other description of your program(s) that may meet the folldwirig criteria: 

1. The program has been^ recently introduced (i.e», no more than four 
years ago) . . . 

■ ■ ' • ' • \ ■ 

2. ^ The program is intended for adults (i.e., 18 years and older). 

3. The program is technical in orientation, not aimed at general 
' ' education, hobbies, or cultu-ral enhancement.'^ 

The program may, of course, involve one or more ^courses. 

We realize that this letter may be misdirected, i.e.,] that someone else 
in your institution may be spepif ically responsible for training programs 
. and for disseminating information about them. Should this -be the case,, we 
would be most grateful for your redirecting this request., 

We hoj^e you will cooperate in making this study possible. 

' Very sincerely yours, 

1 

Samuel Ball 

Research Psychologist 
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, ' ■ ■ APPENDIX C 

LETTER REQUESTING SPECIJ'IC COOPERATION 

I ■ 

EDUCATIONAL TKSTrNO SKR\'rCE ' P R I X C O N. N.J. OSo-tO 

I ^ * ' 

4- - 

Area Cede 609 " ' 

921^9000 ' ^ 



Division 

Educational Studies ' 

\ 

i 



Dear Sir: . 

c ' 

Educational Testing Service, under a contract with the Office of Naval" 
Research, is engaged in a survey to discover current practices in the area 
0 of program evaluation. Our focu5 in on technical training programs for' adults. 

We have compiled a list of literally thousands^ of training programs. 

From this list we have selected your program, , as one we 

would like to study further. * 

What we would like to do is have Ms. Patricia Wright from our office 
call you in the next week to find out whether you are willing to participate 
in the survey, and if you are so willing, to make necessary arrangements 
to have a questionnaire sent to you. . ' . 

Because terms Mke accountability and evaluation in education enjoy 
increasing use, we feel it would be helpful to first determine which formal 
and informal practices and explicit and implicit assumptions occur in the 
real world of program evaluatiorP^and thus this survey. Later we shall 
draw on the results of the survey and on the findings -from other kinds of 
research to develop suggestions and guidelines for those who face the 
responsibility of proposing and conducting evaluations. 

May I thauk you in advance for any help you can provide us in this 
enterprise. , 

Yours sincerely, ^ 



Samuel Ball 

Research Psychologist 
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CASE STUDY . REPORTS 
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REPORT OF SITE VISIT' TO A DRUG ABUSE EDUCATION SPECIALISTS COURSE, SERVICE 
SCHOOL COMMAND AT A NAVAL TRAINING CENTER 



Section, 1 - Program Description ■ ^ ' . . , ■ 

The Drug Abuse Ed^ucation Specialist Program (DAES) is one of several 
programs organized under the "People's Program" that has subsequently 
become the "Human Goals Program" of the Navy. DAES w^s in response to the. 
publicity-'given to drug addiction/abuse by ser^vicemen during the^ Vietnam 
'war. Previous attempts to cope with, drug abuse had been educational 
efforts "to attempt to inform or scare the serviceman through lectures and 
films of the dangers of drug abuse." Reaction to such educational effort 
was that it only created a" greater problem in that sailors who previously 
had been experimenters now became hard and more proficient users. In 
response to these negative observations concerning previous drug education 
efforts, the DAES program was initiated as a developmental effort in 

March 1971. Chief and First-class • ^^ad previously served 

on six-man, drug abuse teams in Vietnam and wer.e two of the rirst 
instructors in the' program. 

From the outset, the program was conceived as an intervention model. 
Drug and alcohol use was perceived as a symptom of other problems, and. 
these prol/ilems related to organizational management as well as to the 
personal /problems of staff! The DAES program was conceptualized to produce 
staff who might become "change .agents'' on return to their unit or command. 
The DAES course had four basic components, which were 1) pharmacology, 
2) political strategy^ and human motivation, 3) communication, and^ 4) program 
development and evaluation. The training was- designed to prepare the DAES 
as a staff consultant with skills to develop interventions and assistance 
to the staff cof the command. The DAES was not intended to be an abusive 
or intrusive intervention but a "requested" intervention by the command. 

.The mission/statement for DAES is "to provide selected personnel the 
knowledge and skill to assist command personnel with the development, 
implementation and evaluation .of drug and alcohol action prd,gi:ams." The 
DAES has seven major course objectives: 

<L 

1. To identify the' role of a DAES specialist at his command. 

2. To demonstrate knowledge of pharmacology of common drugs. 

- 3. To demonstrate personal communication skills with individual and 
group situations. . 

4. To prepare and present learning situations using a variety of 
media. 

5. To demonstrate understanding of issues associated with the use 
and abuse of drugs' and alcohol. 

6. To design comprehensive command drug education and action program. 

7. To identify and utilize appv^opriate resources - 'supportive of 
command drug programs. 
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During the three years of the program, the obj.ectives have been altered 
slightly as more slcill in writing objectives and more pressure to validate.--^ 
^the program have been experienced. The initial training program was four 
weeks, which .was subsequently lengthened ' to . the current five-week program. 

* <? • 

The DAES program, after being implemented for approximately two 
years, produced the observation that there -was need for officer inter- 
vention, in some commands. As a result, the DAPO (Drug Abuse Officer) 
was introduced to provide management training at the officer level of . 
commands needing to establish intervention programs. Subsequently, a 
DAPA program has been initiated to provide technical specialists in drug 
abuse programs, which may provide further assistance to the DAES in the ^ 
management and operation of an intervention program. While this report 
is primarily concernec with DAES, the continuing developments of the 
intervention model has spawilied these two new training programs that 
amplify the initial efforts. ' ' - ' 

This program has been planned to serve every, rank in the Navy. .iS a 
result, enlisted men of every rate and officers have been, found in the 
classes from 197J- - 1974. Most classes have approximately 15% officers 
and 85% enlisted men. Those selected for the school have a GCT score of . 
52 and be released and recommended by *their command 'for the training 
program. Each class (which runs five weeks) has from 20 ~ 35 students. 
A major concern from the beginning has been the atttrition, which has' run 
as high as 40'% in some classes and typically runs 25 '- .30% in each class. 

This program was organized to train 250 men for . DAES billets in the 
Navy. ■ (It was planned that' there should be a DAES for every command 
having 5000 men.) The plan was to train 250 DAES in 1-1/2 years. 
Actually, it took 2-1/2 years to accomplish this training, and with the 
attrition and change of assignment, a continuing need for' DAES and DAPA 
personnel is found. 

The training program has been at the Naval Triiining Center in San 
Diego since 197L but will shortly be discontinued, as the entire Human ' 
Goals Training program, as of March 1975, is being centralized in Memphis. 
In contrast to the 'present five-week DAES training" program, the new 12- 
week Human Goals Program will incorporate a 2- or 3-week module on DAES, 
along wi^h the other elements of organizational development, management 
and planning. . 

There are approximatefy^350 graduates of the DAES, DAPO and DAPA 
'programs. Stude-^ts in the program are evaluat'ed at the completion of each^ 
unit of instruction.- Detailed instructional goals and guides are developed 
for each unit of instruction. Part of the "IG's" are the written, oral 
or observational assessments that are made of the students at completion 
of each, unit or activity. The largest attrition of this program occurs 
in the first and second weeks. At the end of the first week, at least 
half of the attrition occurs from students who drop out of the program 
and return to their unit. The other half of the leavers. are those who 
are unable to handle the "cognitive load" and the intensive study that • 
is required. The pharmacology unit is the first one presented in the 
course and demands the greatest amount of technical reading. The 
instructors also -evaluate the students on their attitudes, awareness 
and sensitivity to the conceptualized role of the DAES Specialist. Some 
students are washed out for lack of appropriate attitude. There has been 
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•a continuing concern with' the'' percentage of attrition; and. although staff 
and external evaluators have -suggested several possible solutions through 
more detailed and relevant selection procedures, it is generally concluded . 
that these selection procedures are impractical because of time and 
communication constraints within the Navy. 

Section 2 - What kind of evaluations were carried out ? 

» ■ 

Program evaluations have been both internal and external. The Bureau 
of Naval Personnel requested that- an external evaluation be conducted . and 
prepared in an RFP, which was sent to "selected consultants or agents." 

. The contract was awarded to the Drug Abuse Training Center, Department 

o-f Psychiatry, School of Medicine, The. purpose of this evaluation 

was to determine. the developmental progress of the DAES school. This 
consisted of a review of previous accomplishments and current operational 
efforts of the DAES cours'e. Ftom'site visitations both to the school and 
the fleet commands, subjective judgments were formed with respect to the. 
accomplishment of the program. The primary task for the evaluation was 
"to determine if the DAES course was doing what it should do, and how well 
it was doing its job." The evaluation *also" was directed at providing 
recommendations for constructive modification vor improvement . of the DAES 
program. To consider the effectiveness of the program, the external 
evaluators 'directed their attention to, 1) program administration, 

^ 2) curriculum, 3) instructional processes and, 4) product validations. The 
latter was done through making a field survey of DAES activities ^^^er^^ 
graduation and assignment'. The title of this external evaluation was "A ^ " 
Subjective Evaluation of the Drug-Abuse Education' Specialist Course, 
Personnel. Management Schools Department, Service School Command, Naval 

' Training Center, " The fact that it Is characterized as a 

"subjective evaluatio'n" reflects the nature of the inquiry, the types of 
information provided, and the reporting style.' The. external evaluation 
was carried out by investigators who were known (by reputation) by the 
staff of the DAES. school. It is believed that the staff's recommendation 
for the awarding of the contract to these consultants had a large part 
in their selection. 

A second external evaluation by the Naval Personnel and Training 
Research Laboratory (not directly connected with the Training Command) 
was an. evaluation of factors contributing to attrition at the Drug Abuse 
Education' Specialist School. This study was requested by the staff of 
the program with some apparent encouragement from CNUT or CNTT as a means 
of developing recommendations for program^ improvement and effectiveness . 
The primary focus of this study was a 6'€0iled .task analysis and tally 
* of. responses of graduates of the school 'i^rogram. This was also paid 
for by the Bureau of Naval Personnel, ai was the external evaluation that 

was conducted by the ^Medical School consultants. The evaluation was 

called "A Task Analysis" by the Personnel Research and Development Center.. 
This evaluation by the Bureau of Naval Research and Development generated 
a unique instrument for making a task analysis, administered the instrument 
to all graduates of the program and completed a statistical^ report of the 
responses of previous students (approximately 65% responded to the 
questionnaire). While no f ormal , report was provided with this task 
analysis, the 'item data were summarized and submitted to the program staff 
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for their use in considering program modification and improvement. The 
design of the task analysis and the treatment of the data appear to have 
been conducted with substantial expertise. The audiences for these two 
external evaluations were in. each case the program funders (Bureau of . 
Naval Personnel) and the program staff of the DAES School. The expected 
outcomes and benefits of the evaluation were apparently largely directed 
toward program improvement; and modification rather than toward decisions 
for continuing or terminating the program. 

An almost continutjus array of consultant/evaluation services have 
been contracted for by the staff of the DAES program. These consultants • 
have provided technical expertise and training to the program staff, in 
every phase of the program. For example, clinical psychologists have 
been used for staff • in-service training on^Gommunications ^skills , 
individual and group work, simulation acU'Vities, personal motivation, 
awareness, values and commitments . /iDther coi|sultants have been used on 
management systems designs and techniques to deyelop "change agents." 
Other consultants have been obtained to. provide models and alternatives 
to drug and alcohol interventj^s . Consultants . have bean used for staff 
in-service concerning the analysis of operational units within the Navy; 
formal versus real chain of command and communication networks, and for 
the identification of alternative strategies of intervention for ■ management 
development and implementation of command op&rations. This array of 
staff-contracted consultant/evaluator s were invited to visit the school, 
make their observations, and then be used by the instructional ■ staff of 
the school to "pick their brains" for content, process" or organization 
that might improve the motivation and effectiveness of the program. While • 
these consultants were paid for by the Bureau of Naval Personnel (at 
the request of the training staff), no formal evaluation reports were 
provided. In general, the repotts were the consultant's documentation 
of the activities or services he provided to the DAES staff during his 
tenure of consultancy. It was the anticipation of the DAES staff that 
these consultants would provide them with new knowledge, content and 
insights for their own personal development as well as program improvement. 

• Their perception of the vusefulness of these consultants was directly 
related to their combined judgments as to whether the consultant was _ 
providing useful input into their ^program or the processes that they might 
modify of improve. / 

It, should-be .stressed that the evaluation of the DAES program, was 
directed "heavily toward the monitoring of ongoing operations, the analysis; 
of contents and processes, and the suggestions for improvement of program- 
effectiveness. A classic and comprehensive evaluation design for program 
evaluation was generally lacking . No comprehensive model or ' application 
for program evaluation was found in any of the evaluation efforts. This 
is thought to be related to the fact that DAES was a new conceptualization 
that was undergoing almost continual modification with each _ succeeding 
class. .-Specific objectives and processes were also undergoing 
modification. 

When the staff were interrogated as to the model of the external 
evaluations^ they responded that they were not aware of any model 
or. design of evaluation. They expressed . great confidence in^ the 

• consultants and respected their subjective approach, to assess the school 
and 'follow up with the graduates in the .various fleet commands. Typically, 
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the consultants returned- to the school and informally shared their 
observations and evaluations of content and process with the instructional 
staff. The DAES staff, to a man, reported that the evaluation contributions 
had been very helpful to them. They believed the' most help was the almost 
continual sharing between evaluators and staff in the course of their 
assessment and evaluation activity. A substantial purpose- t>f.. the program 
staff of the external evaluations and employment of consultant s/evaluators 
was to obtain documentation which would justify the attendance of the 
training staff at educational conferences, workshops and other forms of 
formal and informal training. For example, the staff have attended 
workshops on transactional analysis and NIMH evaluation workshqps put on 
by educational institutions and have made contact with, or matr'iculated 
in, formal courses in higher education with relevance to their intervention 
program. The director and staff of DAES believe that the -subjective 
evaluation reports of the external evaluators and the employed consultants 
provided at least 50% of the documentation which justified increased 
appropriation from the -Bureau of Naval Personnel to f und inservice training, 
other forms of education and wide visitations by the program staff. Firom 
thl.a standpoint the staff believed the evaluations were extremely successful 
and accomplished their goals or purposes. In contrast, it is questionable 
whether explicit documentation of. the operational effectiveness and the 
validity of the operation from the standpoint of. the Bureau of Naval 
Personnel was provided in the somewhat subjective reports of both the 
formal .external evaluators. and the consultant/evaluators employed by the 
staff. Thus, on one hand, the evalu^^tions were perceived as effective by 
staff, but it is unknown whether other recipients of the reports (CNET, 
CNTT, and the Bureau of Naval* Personnel) share the value of these reports. 
The only inference that may be drawn is that the Bureau of Naval Personnel ^ 
increased the funding of this program and' inservice training for the 
instructional staff. « v * 

In the classical sense of systematic and comprehensive models.of evaluation, 
these evaluations left something to be desired; they would certainly be 
characterized as more subjective than systematic, and the reports must be 
characterized more as impressionistic than documented by systematically 
acquired data. For example the task analysis conducted by the Navy Personnel " 
Research and Development Center made no mention of the disparity between the 
number of questionnaires distributed and the number of questionnaires 
summarized. It is .believed there was something close' to 25% attrition from 
distribution to cpmpleted -questionnaires. In light of the extremely sensitive 
nature of intervention programs, it would appear highly important -that this 
attrition be dealt with in submitting an analysis of the task analysis data. 

In addressing the question of the level of technical competence with which 
the evaluations were conducted,, one must consider the difference in design of 
evaluation for ongoing developmental programs and those that have established 
dimensions, goals, objectives^ contents and processes. ,, This DAES program was 
kaleidoscopic in its almost ever-changing manner. Thus, the conventional or 
traditional program evaluation model would probably be ill-adapted to this 
situation. At the same. time, it appears that there was substantial lack in 
identifying the goals and objectives of the course presently existing as they- 
considered the effects or outcomes on the student graduating from the course. 
No specific assessments of skills or interventions were undertaken for the 
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DAES as he returned to his command. All of the assessments seemed to be 
observing what the DAES was doing rather than assessing to what degree 
certain anticipated outcomes w^e being achieved. Thus, one might say that 
the technique for assessment in these evaluations was more concerned with 
providing descriptive information of what existed in a fairly open-ended 
manner, and thus providing opportunity to obtain pbs.ervations about unintended 
as well as intended results from the program. . 

Perhaps one of the most serious concerns relates to the fact that tJie objectives 
for the course were not always fully understood and communicated to the command 
from which the students were coming,^ It was also concluded by the staff that 
the skills of the DAES as a change agent could not be overtly assessed in 
conventi9nal terms since the success of the "change .agent" would be reflected 
by his lack of visibility or lack of. obvious efforts at making changes in 
management and command procedures. Thus, there was substantial ambivalence 
as to whether discrete skills, knowledges and actions were appropriate 
exemplars of success for the eight major objectives of the school and the 
larger mission objectives of the DAES. 

Some technical competence was demonstrated by the Naval Research Laboratory 
in their, analysis of the Strong Vocational Interest Test and an attitude 
test specially constructed to test students in the school. These instruments, 
were administered to all entering candidates; and^ at the conclusion of the 
class, blind analyses by .the~Naval Research Laboratory were made of their 
predictions of those who would be successful and those who would not. It 
was found that the profile of the Strong and attitude tests combined would 
offer predictions at the 90% level o^ accuracy. 

A good illustration of the lack of use of evaluation measurement data is the 
fact that while attrition was the' greatest problem to the program and there 
were solutions suggested offering preliminary validation, other problems 
were cited as overriding in using such assessments (time constraints, 
communication problems, and the, mobility of the Navy personnel). Thus, while 
some assessment and evaluation activities were conducted, with rigor and 
technical competence, there was no commitment to the notion that if findings 
suggested more desirable practices, changes would be made. A" commitment for 
actions or uses of evaluation that might be made was obviously not made prior 
to the contract or conduct of the evaluation. Another example is the creation 
of a "learning style inventory" that was administered to all of the classes ■ 
and was. found to be extremely useful in allowing individuals to develop 
insight into strengths, weaknesses and* par ticular emphases that would improve 
an individual's effectiveness during the course of the program. This 
procedure^ however, was apparently not routinely used and became only a 
providential addition as time and circumstance might allow. 

One of the greatest paradoxes of ;this, program evaluation is the fact that the 
instructional staff, the director of the program, and their continuing 
consultants held different criteria fpr program effectiveness than the 
criteria .selected as co.nventional indices by the Bureau of Navaf Personnel. 
Thus, the- Bureau will probably require hard data for the judgment of 
•effectiveness that will deal with such things as fiOPC retention, court 
martials, discipline, ' advancement records and incidence of drug and alcohol, 
abuse. In contrast, the staff believe the real success of the prograin is o 



in relation to developing morg effective management and development plans 
in each command so that , organizatidft and management of coinmands carry out 
the human goals program. The staff believe these outcomes, 'however, may 
not be readily observed, and they do not believe that the Bureau of Naval 
Personnel or the commands .would put up either the budget or the time to do 
a sufficiently thorough,' relevant and reliable evaluation of these outcomes. 
In contrast the more simplified statistical reports on incidences . of various 
kinds of drug and alcohol abuse, retention, etc., will be usedo as evidence 
of success. f 

The staff evaluation of the relative effectiveness and usefulness of the 
several evaluations was extremely positive. Directors, coordinators and 
instructors were in agreement that -the ^evaluations were subjective reports 
but were extremely helpful because of- the nature of the consultants who ^ 
.provided the subjective reports. Obviously the credibility, acceptability 
and facility of communications became substantial elements in the perception 
of tAe DAES staff of the effectiveness and usefulness of the evaluation. 

Two consultant/evaluators were cited as individuals who lacked awareness of 
the conceptualization of the program and provided mechanical or "canned" • 
solutions for program modification and improvement. These consultants were 
quUckly rejected by the staff, and the consultant/ evaluator contract was 
---terminated. 

The program represented an unusual developmental effort by individuals who 
represented an openness to analyze and^ grow both in terms of ^personal attributes 
and program development. As a result, their-interpretation and evaluation 
of the consultants and the evaluations^. may well have*been in relation to that 
which was perceived as useful and meaningful for the particular stage- of 
development pf the staff and might well be perceived • differently by others 
with less open and inquiring attitudes.. A staff with^constrained- views of 
what a program should be and with great interest in setting in concrete the 
program with specified contents, dimensions and outcome objectives would 
probably appreciate different evaluations. This group represented a free, 
open-inquiry approach and as a result were extremely appreciative of the 
subjective evaluation reports. .The consultant/evaluators had high credibility 
both personally and professionally with the project staff." These subjective 
reports undoubtedly held some credibility for the Bureau of Naval Personnel, 
as reflected by the very extensive and adequate funding of these programs. 

The staff did agree, -however, that- outcome statistics would have been very 
useful. They would like to use outcome statistics with. their students to ^ 
illustrate the needs of the programs. 



Th^ staff clearly identified that a critical analysis of needs data (of 
statistics, demographic characteristics and attitudes) would be extremely 
important for designin'g such a pro^^ram. Such a needs analysis was not 
conducted, and as a result they felV-ther-e was some lack and some necessary 
hesitance or staggering around to finally develop a. match between the need^ 
of commands, the needs of persons in training, and the needs as perceived by 
the Navy for improving their human goals program. On the other hand, the. 
subjective reports made to the Bureau ""did have some impact. In at least 
three* instances, the Bureau demanded that a certain kind of, training be 
emphasized and carried Dut. While the staff of the training school did not 
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fully agree, because -of / Bureau insistence, they modified tHe program as 
requested by the Bureau. For example, there were several types of 
experiences which th/ staff felt would be helpful for the students, such 
as visiting the ghe^tto and contacting drug pushers, etc, -Such practices 
(reported in the evaluation reports) were so offensive to higher-echelon 
connnand and Bure,du personnel that by insistence they were terminated. Thus, 
one would have to conclude that the evaluations did indeed effect funding | 
and program mp'dif ication. I j 

One would have to conclude that, in anticipation of a formal complete model^ . 
these eval'uations fall short of the mark. They did not make systematic \ ' 
assessments of either process or product tut wove an interesting fabric froin, 
the subjective evaluations of the program' s goals, objectives, personnel, / 
processes and contents, and the observed actions of people after they had . '/' 
completed the program. 

The DAES program is completing two more c^lasses/ and will be moved as^ a 
sub-element in the, new Human Goals Management Program housed in Memphis. 
This consolidation of training efforts is the Navy's attempt to develop a 
human resources management center in Memphis, where instruction; in all 
aspects 'of the human goals and human relations as a management problem will 
be housed. In "this regard, the new Navy Human Rp.source Management Survey 
is perceived as ^ an instrument which has borrowed from> the- observations and ' 
developments of the training com[mand's exper ier.ce.. The questionnaire will 
be useful' to the DAES and any number of other command personnel in working 
on human relations as a management problem. 

The special lessons that might be learned from studying these evaluation 
>efforts have probably been referred to previously, but in -summary seem to 
be the'^lfollowing : ' ' . ^ 

1. Innovative and developmental programs probably need less restricted 
or constrictive evaluation designs and models than are typically 
applied to'-static programs that are to be evaluated in relation to 
cost-effectiveness, validity or productivity. 

2. Evaluations, regardl,ess of model or design, constantly interact with 
or are confounded by the nature of the personnel who are. being ' 
assessed, receive the reports, or must make decisions.. -The nature 
and expectations of the key personnel involved'in and affected by , 
evaluation are frequently nbt known by the evaluators or are not 
known by the people. who ate requesting the 'evaluation or .are to 
receive the .report. However, without 'knowledge of this, it is not 
infrequent for a mismatch ^ to be found betweeA the nature of the ■ ,^ 
individuals intimately involved in the assessment and the evaluation 
and the various external evaluators. . ConflictV in^ attitude, values, 
terminology, and communication style may become so critical that 
the assessments may be aborted; and the ultimate evaluations, 
whether subjective or documented, may be ignored as irrelevant. In 

, 'contrast, where empathy and. high-fidelity communication exists 
between cpnsultants/evaluators and personnel being assessed, there 
appears to be high likelihood that even somewhat critical and 
derogatory statements may be taken for self-improvement . rather 
than rejected as self -destructive. » * '■ ■ 



Needs analysis prior to. program development and implementation 
and/or- evaluation continues to be an often ignoredvbut ultimately 
important aspect of the evaluation process. The "is-'should'^ 
model appears tb be a useful one that is easily understood- by''a ' 
staff relatively unsophisticated in evaluation. The "is-should*' 
model and the management development systems model appear to be 
useful in -helping staff develop insights' into the nature of 
assessment and evaluation.' After the fact, the staff discovered 
the Importance of the needs assessment arid lamented that it Was . 
now "too late. " , * • t ' . 

The effectiveness' of this evaluation was highly related to the 
personality characteristics, of the principal staff members* engaged 
in the operation of the program. The. openness of the staff may 'be 
directly related to the conclusion that the ' evaluation was useful 
and effective. One may question whether summarizations " about the 
relative effectiveness or usefulness of program evaluation, may be 
as much a function of the attitudes and characteristics^ of the 
recipients as of the technical competence, comprehensivenes's or 
effectiveness of the evaluation conducted. 

It has long been stated t]hiat it. Is frequently difficult to find ' 
a constituency for evaluation. ' Yea, though I spe^k before many 
groups, P rarely find th^ clamor, "Please come, and evaluate me!" , 
The enthusiasm for evaluation seems to swell largely in the 
evaluator rather than th^ evaluatee. The antidote for tl^is 
generalization may be fdiind in the involvement and maturity ^qf 
the various persons who ^re intimately related or effected by 
the evaluation. - , ' . , 



REPORT OF/mTE VISIT TO*u!' S. SIGNAL CENTER AND SCHOOL CONCERNING A 
ipiGITAL sMsCRIBER TERMINAL EQUIPMENT REPAIR COURSE' 

Introductidn ' ' 

: — : \ — \ , « ■ y 

In^ir^Sding this repor t ,and ' the program evaluation questionnaire it 
is important to note that the elements of evaluation described are part ^ 
of a coinprehefisive system of training and evaluation which transcends the 
speci^ac course heading of the qiiestionnait^ and this report-. Neither , 
the specifics of the questionnaire no-r the content of this report can 
ade.qjiately reflect what is^ considered by this' reporter to be a most 
efficient and thorough-going ef for t . The report will, however, attempt 
to ydescribe^the* more general system at~appropriate places sa as to put 
fhas specific course into the brdade.r evaluation context and to qualify . 
p/roperly the more. sp€^cific information -reported in the questionnaire. 

^The. P-rogram ' - > tj [ ' • 

■ - ■ 1 
The U. S. Army Signal Center and School at _j_ i- 



. is one of a^'natwork of twenty-three training schools lo^cated in various ^ 
^parts af the United States. Although the schools differ in the types of 

training offered, their ''operations are based on common -policies , regu^ i* 

lations and pro'cedures^ developed under the. broad review and direction of 
"the United States Army VTrainirig and Docttane Command (TRADOC) at 

' Virginia./ . . 

, *i ; ■ ^ ' ■ 

The repair course described in the Program Evaluation^-Ques tionnaire 

■for this study is one of over thirty courses offered at - to. 

•some 15,000 trainees a year inpludiag small propprtions of trainees fron^^ 

other , service branches whose military occupation specialties (MOS) deal; . 

-with milit'^ry^ communications and electronics . ^ 



, 'Although the training-' pr^ogf am includes instruction for commission^ 
officers, the majority of- courses are provided for nori-commissipned 
.officers and^other personnel .in selected. military occupation- special ties . 
These -courses typically include "hands on" training," center around quite 
Specific performanc^e objectives.,, and usually' require some- 15 to 20 weeks 
to- complete in peacetimje, ^ Some course^^afe offered several times in a 
calendar year. .Each course, \ori/fevery occasion ^offered, i-s^ubjected to 
the same evaluation procedures.- . 

The Evaluation ' ' / ' ■ ' 



Very broadly viewed, the evaluation of this program is included in . 
the seventh' and last majot 3tep of a- formally developed model referred to 
as '*^S^5^ms Engineering o^ graining". The major steps in thi3 model are: 

'<yob anal3^sis, selecting ta.sks for" training, . training analysis', developing 
training material,' developing 'evaluation materials, conduct of. training, 
'and quality control. Sy/ste^n engineering,- became required fo.r all courses 
in 1973- via. a major document on the suBj.ect issued by TRADOC, At tbigt 
time a major effort .was /mafe to eliminate or greatly reduce theory-based 

' 'irid true t ion and testing in favQ^--©-f-.;perf p.rmance criteria and/ per^ormaiice 
testing. " '• • ' 



In the spirit, of the new directions, the major goals of evaluatiion 
are: 1) the certification of ability to p&xform specific, tasks at the 
termination, of the course, and 2) the provision of feedback for purposes 
of course^ improvement . Of course, each course cannot be changed by small 
increments for each new class; rather the results from each 'class evalu- 
ation are considered together when plans of instruction are periodically 
changed. v 

The overriding value from- which both the model and the nature of 

measures are derived is explicitly stated as follows: ''training objectives 

will be based on job tasks and tests will be based on the criteria estab- 
lished for the training objectives'^ 

Again in general terms, it is intended that in quality control "all 
elements of the instructional system are examined and adjusted in order 
to assure that the desired qufvlity of training is achieved with the 
minimum expenditure of resources". 

Both internal and external sources of data are gathered and analyzed 
formally. Informal evaluation efforts are made, as well. These procedures 
are described in some detail later on in' this report. 

Evaluation in the Array's schools, as indicated in the Introduction, 
is part of a comprehensive system. As such we would, expect it to be more 
formal, more regular, and institutionalized to a greater extent than 
efforts elsewhere described as less systematic. And so it is. 

Evaluators are fuil-time professionals occupying forinal positions in 
Evaluation Branches at each of the schools. These positions are provided 
according to procedures described in a staffing guide foi*' Army Service - 

Schools, In the case of the school at , th^ Evaluation 

Branch consists of eight full-time professionals with vatied experience 
whose chief is a senior civil servant with over thirty y^ars experier^ce 
in evaluation at . . . ' 



The evaluations required of all courses at the scho 
with funds formally budgeted- for the purpose by each sch 
by commands both locally and at TRADOC, These budgets ±tic 
staff, materials, data processing services, etc. 



Is are conducted 
ol and, approved 
lude all costs • 



Although from time to time interest in the results 
course evaluation extends as 'far as field commanders in 
,the primary audience for course evaluation results is th 
at each training, school,. Since TEADOC at 



of a particular 
overseas posts, 
e local command 
IS responsible 



for coordination and siipervision of all training for the Department of 
the Army, their group, too, is a recipient of all reportis of results. 

In general terms it was expected that this course evaluation, like 
those for most courses, would "verify that the course wis producing 
graduates capable of performing their tasks in on-the-jcpb situations, 
and contribute specific information for .course improvement". Expectations 
held for this course evaluation were realized. Based on analysis of the 
data from questionnaires , and ratings the report concluded that '-'the 
course did effectively prepare graduates to perform competently when 
assigned to fi^ld units" and indicated as well two .^reafe of course work 
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where additional time was needed. These findings were anticipated since 
the course had been. offered several times before, had been "systems- 
engineered-' and had been approved in revised form by TRADOC the year 
before. 

Evaluation Methods 

The formal evaluation might be described as -a two-stage strategy. 
The first stage concurrent with the presentation of a course consists 
primarily of performance-based measures often consisting of test situ- 
ations set up on actual equipment, or utilizing simulators or mock-ups, 
when circumstances prevent the use of the actual equipment. Detailed 
procedures for direct testing of this type are required and specified 
in such documents as .the revised edition of Performance Testing for 
System Engineered Coursed prepared by Evaluation Branch personnel. 

Evaluation we have termed "concurrent" for purposes of this repo 
is intended to assist instructors to locate strengths and weaknesses 
during instruction and to verify that .trainees meet course objectives 
their satisfactory performance in the test situations. Such sources 
referred to as internal sources o"f quality control data by the Army. 

The second stage focus and the more important criterion for success 
of individual trainees and the course is performance on the job. Assess- 
ment of performance on the job is delayed until trainees have been on 
duty approximately five months. Evidence is then gathered from two 
sources: from the graduates with elaborate questionnaires and from their 
supervisors through questionnaires and supervisors^ ratings. These 
sources are referred to as "external" sources by the Army.. 



by 
are 



(We should note here that the first stage, with its attention to 
very careful specification of steps and procedures from original specifi- 
cation? to methods for equating instructors grades, is omitted from the 
program evaluation questionnaire.) 

Info'rmal evaluation is also undertaken. Internal sources of less 
formal data include audits or observations, student comments and 
recommendations, and of course, the comments and recommendations of 
experienced staff and faculty. 'Examples of external sources of less 
formal data are reports by commanders, combat reports, field test results, 
and interviews with returnees from field assignments. 

Alt.hough the specifics of the performance testing for this course 
were not observed or discussed during the visit, the- fprmal external 
source instruments 'can be described briefly. 'tThe postgraduation 
questionnaire was administered five months after training was completed. 
It is job-oriented, requesting twelve pages of detailed information, in 
this case on 203 job tasks, organized around some nineteen headings such 
as Site Operation and Maintenance, Receiver Site Equipment, and Equipment 
Repair. . - 

/' 

The Supervisors^ questionnaire typically covers questions about the 
graduate^s duty assignment , and on-the-job training and provides ratings 
of his job proficiency in major task areas as well in skill and 
knowledge areas. Standards and Definitions of the ating scale positions 
are provided to contribute to improved reliability of the ratings. 
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The tvo questionnaires are mailed to the commanding officers of 
appropriate units with a covering letter from the Commanding General of 
the Signal Center, and School. Reminder cards are attached to both 
questionnaires requesting the students and the supervisor to return- them 
completed within seven days. 

>> 

Evaluation Results and Effects 

Qualified full-time staffs in Evaluation Branches working I'rom 
detailed specifications and procedures almost guarantee that evaluations 
such as these will be carried out with technical competence^ 

It is important for the reader ac this point, however, to distinguish 
clearly between evaluation and research. These distinctions are far 
from superficial or pedantic. There are" major differences between the 
two, and military training is a noteworthy example of the utility of 
viewing them differently. One distinction relates to the purpose for 
'^the analytic activity - a situation which centers on the outcomes of a 
very carefully developed plan of action, (evaluation) as distinct from 
general and open investigations (research). 

Another distinction is the usual absence of 
variables, comparison groups including controls, 
quasi - experimental designs. 

The logic of the system, engineered course and its analytic pro- 
cedures result in the view that what is offered is the best under the 
circumstances; all trainees are to receive the same program because it 
is considered t^e best under the circumstances, and the overriding 
purpose is to produce as many with the proficiencies indicated as are 
exposed to the training. It must not, it cannot, be withheld for 
' purposes of experimentation. .^Experimentation and research, while impor- 
tant, are reserved for other situations with more uncertainty. 

The main point of this apparent digression is to note that the 
more traditional research - oriented elements such as variables, designs, 
etc. are often inappropriate indices by which to judge an evaluation. 

When judged from 'an understanding of the necessary distinctions 
between evaluation and research, this evaluation and others conducted at 

are especially noteworthy examples of competent evalu- 
ations which succeeded in accomplishing their goals. 

The evidence on the matter of accomplishment is varied indeed - in 
this case, the objectivity provided by competent professionals in the 
Evaluation Branch, comments from a -review by the Department of Data 
Communications, concurrence with the comments from Data Communications 
by the Curricula Branch are examples of the multiple reviews this 
evaluation received. All evaluation studies receive these multiple 
reviews by independent units. 

at y New Jersey were quick to point out, also, 

action at the command level to implement recommendations from 
studies was the best evidence that the evaluation accomplished 
It also serves to underline at the command level the 
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specific experimental 
and matters of true or 



Staff 
that quick 
evaluation 
its goals. 



/ 



importance that is attached to evaluation by the Department"^' of the Army. 

Although there appears to be no formal provision for or concern 
with unintended outcomes, which lie outside the course objectives, or 
side effects of the program, -^he informal sources mentioned earlier are 
probably varied enough to detect serious side effects should a program 
produce some. 

Overall Chara.cter-is tics of the Evaluation 

The absence of Methodological . and operational problems in this 
evaluation important enough to surface, either in an on-site: visit or 
in "reviews- of major documents and reports of the program, is the direct 
result of the role evaluation plays in the Army training Schools. It 
is a routine, and like all routines, the more often it is performed the 
smoother the operation becomes. 

This reporter would speculate that some quantitative specialist 
bent on finding some methodological flaw in the thirty pages of tables 
and narrative would be rewarded; he-might point out, for example, that 
apparent differences in proportions were not tested for significance 
or, as mentioned before, some effort to apply quasi - experimental 
design would "^stre^igthen the report^ or that additional correlations 
between course grades, performance tests , supervisor ratings, etc. 
should be undertaken. 

In the opinion of the writer, however, this would be missing the . 
point. 

There are several features of this evaluation which, when interwoven 

as they are here, make this effort noteworthy: 

- the course objectives, the course material, and later performance 
on the job deal with specific behavior which can be readily 
specified and observed; 

- the systems engineering of courses . s a carefully and logically 
developed backward reconstruction of important analyses begfin- 
ning and ending with analysis on the job which contributes 
tangible results and documents all along the way, and which can 
be refined further in successive efforts; 

k 

^ ' - the procedures for instructional delivery and for evaluation are 
• carefully developed, presented in written specifications, and 
implemented as specified there is an almost total absence of 
" ambiguity or irrelevance or redundancy; 

- the measures of the results of instruction are extremely potent 
performance measures offering instructors valid and direct 
evidence of success; - ^ ^ 

- the confi-dence in the use of evaluation results by commanders is 
justified not only from previous experience but from the mutual 
respect between line officers, and dedicated civil servants who 
lend professional stability to the enterprise. _ 
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Key ideas in the evaluation can be represented by the words 
"systematic", "documented", "feedback", "performance criteria", and the 
military term "command action". 

While it ±s recognizJed by this writer that many programs in public 
education do not lend themselves as readily to the precision in specifi- 
cation as this and other training programs do, nevertheless in his 
opinion there is no better operational example anywhere of the education 
al principle that evaluation is a part of the instructional process. ^ 
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REPORT OF SITE VISIT TO AN AIR FORCE BASE CONCERNING THE PROGRAM 
FOR TRAINING AIR TRAFFIC CONTROLLERS 



Section 1 - What is the program like? 

This is a program to rrain air traffic-controllers to serve for the 
Air Force. Program serves approximately 600 to 1000 students per year. 
Persons coming into the program are" either nonprior service personnel 
or personnel being retrained for the position of air traffic control 
operator.' Major criteria for student selection is an aptitude, j*basically 
an electronic aptitude, of the 80th percentile or higher. The goal of 
the program is to train air traffic control operators to a point where 
they can then go to "on-the-job" training. Major criteria for success 
in the program is completion of FAA requirements for position of air 
traffic control operator. Students are evaluated primarily with 
criterion-referenced tests based upon FAA guidelines. 

Section 2 - \^at kind of evaluation was carried out? 

The evaluation for this particular program is primarily a feedback 
.from people on the job, either graduates of the program or the supervisors 
of the graduates of the program. The basic goals of evaluation are in 
the area of curriculum revision. They are not involved in the direct 
evaluation of students nor are they involved in the direct evaluation of 
instructors . . 

The evaluation report we received is one of some 40 course evaluations 

per year generated by the Training Evaluation Division at AFB. 

Every evaluation follows a common procedure: 

Statement of goals for the evaluation 
Sample selection 
Mail questionnaire 
Field visit 

Staff evaluation of curriculum 
Analysis and report 
Follow-up 

The backbone of each evaluation is a questionnaire including a 
checklist taken from "Specialty Training Standards" (STS) . The STS 
defines the tasks expected of a person in a specific job. This is 
perhaps one of the best examples of criterion-referenced measurement 
I have seen. These detailed, task oriented criteria make the evaluator's 

job much easier. In the words of Director, , "Our 

evaluations are primarily designed to determine if graduates are able to 
perform the task on the job at the stipulated proficiency level." 

The paper "Evaluation of Courses Developed Under Instructional 
System Concept" is an excellent presentation of the philosophy and 
procedures of course evaluations at AFB. 
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Section 3 - Who carried it out? 



The evaluation was carried out by the Program Evaluation ^Division. 
This division consists of a director, seven training specialists, and 
four support personnel. These people were rotated in position of / 
evaluator; they were formerly administrators in the instructional program. 
They are basically in-house evaluator s, however, they are not in the 
direct chain of command for the instructional program. They answer 
directly to the commander, of the school. 

^ ' .: 

Section 4 - Who paid for the ,evkluation, how and under what circumstances? 

This evaluation is conducted from the command level (personnel are 
budgeted at the command level) and there seemed to be no problem with 
"finances in the program. 

Section 5 - Who were the audiences of the evaluation? 

Audiences for the evaluation are mainly the school commander and 
the instructional program. , ' • 

S ection -6 - What were the expected outcomes and benefits of the evaluation? 

The benefits of this particular evaluation are in the area of 
curriculum revision. As stated earlier, this particular outfit is not 
involved in the evaluation of students or the evaluation of instructors. 
Their main job is to go out and survey people on the job who have 
completed the program to determine whether or not the training they 
received was adequate preparation for the job they' are now performing. 
Based upon this evidence, they then make recommendations for changes in 
the training program. 

'section 7 - What did the evaluation comprise of? 

The evaluation was primarily based upon a mailed questionnaire to 
either, graduates of the training program or supervisors of the graudates 
of the training program. 'The backbone of this questionnaire is a task 
checklist for the job to which the graduate has been assigned. In addition 
to this mailed questionnaire, the evaluators also conduct on-site visits 
and interviews with former graduates to determine the strengths and 
weaknesses of the training program. 

Section 8 - Was evaluation carried out with- technical competence? If not^ 
what went ^wrong? <? - 

This program appears to be carried out with a great deal of technical 
competence. The outline of the evaluation in terms of planning an.d in 
terms of design is very good. The people who actually do the technical 
evaluation are not trained in evaluation but are' rather subject area 
specialists who are familiar with the training aind" the job to which 
graduates will be sent. The evaluation plans as mentioned previously 
are very much -structured. 
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Section 9 - Did the evaluation accomplish tl>e evaluation's goals? 

Yes, most of the people here seemed to be well-pleased with the 
evaluation results. The instructional personn^.l indicate that the 
recommendations aire in a form which they can^use to make revisions in 
the instructional program. The commander of the school 'lias often given 
credit to the evaluation unit for their excellent work.-. 

Section 10 - Did the evaluation seem to help the audience for whom it 
was intended? Were any changes made as the result of the evaluation? 

Yes, there were some examples given of changes made in the 
curriculum, particularly in terms of the type of equipment used in the 
training, the amount of training time spent on various tasks, and, to 
some extent,' changes in the methodology used by specific instructors. 

Section 11 - Did the evaluation look for unintended outcomes? 



Only superficially.' There were some open-ended items on the 
questionnnaire that asked the gradual:es or supervisors of graduates to 
indicate areas, where there was over-training or areas where training " ' 
was needed and was not provided. The evaluators attempt to follow up 
on open-ended responses in on-site interviews. 

Section 12 - What was good about the evaluation? Was there ajiy thing, 
quite noteworthy? 

This is an excellent evaluation effort given the circumstances under \ 
which these people work. The strong points of the evaluation are (1) that 
the evaluation effort has the excellent support of the commander of the 
school-,, and (2) the personnel involved in the' evaluations - are subject 
matter experts so, that their credibility is quite eye-winning when 
evaluation results are reported back to instructional personnel. This 
program obviously has an advantage in- that they are able to keep track 
of the personnel when they leave the training. The evaluators are . 
able to trace these' people to the job 90 days or even a year after their 
.training. They have tremendous control, in that if they send out 
questionnaires, they go out as a directive and the perspns involved are 
required to return them. Consequently, they consistently get usable 
return rates of greater than 85%. . ' . * *t> 

Section 13 - What 'were the problems with the evaluation? 

There were no really obvious problems that I could find in the 
inter-^ews that' I conducted. There was mention, at one. point, that all 

"the instructional personnel did not take kindly to the evaluation reports. 
Some resisted the evalua^tlon unit more than others. Two potential -..^.^ — 
problem areas are tliose that were listed as strengths of the .-program. 

^Namely (1) thp. concern of the commander of the school and his support 
for the .evaluation effort and (2) the credibility of the evaluation . 
staff. The type of evaluation being conducted here requires a great 
deal of credibility on the part of the . evaluators . 
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Section 14 - Are there special lessons to be learned from studying 
this evaluation? 



I think that the particular lesson that mighpbe learned from 
studying this evaluation would be first' of all tihat a great deal of 
research design and complicated statistics are not needed tor an 
effective evaluation. Kost.of the reports generated from this -unit 
report, p^^haps percentages and frequenpies, but rarely go into 
statistical tests of significance. There's a philosophy here that 
the evaluator should be separate from the decision-maker and that 
is the./way the program is structured. This program has a' turnover 
rate'for evaluators of . approximately three years. The program is 
structured such that new personnel can come in, and take over very 
"quickly. Actually *the deci^sions to be made are' outlined in great 
detail and the amount of structure to the program allows personnel 
to come and go without, ttfo much effect. The philosophy of tliis 
evaluation stresses the practical criteria of the evaluation Hata., 
•such as relevance, . importance, ls.cope, credibility, and timeliness, • . 
and emphasizes these criteria over criteria of reliability predictive 
validity, etc. There appears to be an excellent rapport between the 
evaluation unit and the instruction unite This might be attributed 
to the fact that oral reports are quite common with the evaluation 
unit often reviewing their findings imnjediately with the instructional^ 
personnel, long before it is put in wi'iting. Instructional personnel 
are also included in the .site visit team so 'that they can get a 
firsthand look at the program as it operates in zhe field. 



D-19 



REPORT OF SITE VISIT TO A MISSILE AKD MUNITIONS CENTER AND SCHOOL TO- ^ ^• 
INVESTIGATE THE EVALUATION QE A RADAR REPAIR PROGRAM ... 

The Radar Repair Program, is one of approximately 70^ 

courses taught at MMCS. It is a 25~veek program^ for enlisted servicemen, 
many of whom have had no prior' service occupation assignment. Graduates 

of the program may be assigned to installatiop^s in Alabama, 

Florida, Texas, Germany, or Korea, Prerequisites for the course' inclnide 
full use of both hands, normal color perception, and' an electronics' 
aptitude at or above the ^Qth percentile for all recruits,- The course ^ 
is taught thrSHigh classroom instruction -and lab/workshop practice with 
maintenance tools and equipment. The course qualifies graduates to 
inspect, \test, and repair the ^ * A cquisition' Radar s^s.tem ,and 

related components. - " * 

The major source- of information for the evaluation of the 

program and other courses taught at MMCS i,s feedback solicited , from 
graduates of the course after their assignement to . an operational 
ixnit. Each graduate is asked to complete a maired questionnaire three-five 
months after .assignment . The questionnaire is based upon a task analysis 
Which is • the backbone of the training curriculum. The graduate 'is asked 
to indicate how. of ten he is required to perform each t'ask in his 
"assignment and to comment on the adequacy of his training for" each task.' 
The average return rate for these questionnaires is abouB 60%. 

Two. additional sources of feedback are also utilized: Rotation of 

instructors and on-site visits. Instructoir s f or the program are 

servicemen who graduated from the course and have served a tour of duty, 

as a repair technician. When new instructors are rotated to 

MMCS from the field, they are extensively debriefed with respect" to 
"actual job requirements and modifications needed in the trairiing program* 
Periodically, a team from MMCS will, conduct a tour of bases receiving 
■graduates of MMCS courses. The purpose of these visits is to obtain 
a better understanding of the environment in which the graduate will be 
working, and to hear comment's, on the adequacy of ti^ining from a^variety 
of staff positions. Since many of the bases receiving graduates from 
MMCS are located overseas, evaluation tours are conducted only eT?:ery 
2-4 ye.i'rs. 

The MMCS school is divided into three major divisions: A Training 
Development Division, a Training and Education Bivision, and a general 
support. division. The Training Development Division (TDD) has 
responsibility for the development and updating of task analyses which, 
in turn, serve as the content base for courses taught at MMCS. The 
Training and Education Division (T&ED) has the responsibility for the 
^ctual conduct of training. This division plans, and implements 
methodologies for content specified by the Training Development Division. 
The Training Developm'ent Division also designs -measurement instruments 
for MMCS courses, but T&ED makes decisions concerning acceptable student 
standards. 
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The mail ,sul.vey program and the site^ tour*'program are both. Training 
Development Division activities. An-* annual report based, upon the 
findings of these activities is submitted to the Training and' Education 
Division.- Both divisions make extensive use of returning instructor 
debrief ings. ^ * - 

The Training Development Division consists of 58 persons with 15 
persons devoting full or substantial time to task analysis and course 
evaluation based upon ^graduate feedback. The director of the division 
"and most of the professional^ staff have graduate 'training in educational 
psychology, instructidnal' systems , or some related field. Most of the ^' 
professional positions are filled. 'by civilians. 

The greatest weight in curriculum development or revision is given 
to the task analysis conducted- by technical experts and persons who 
have performed the' job in the field. The mail survey is used primarily 
to confirm this task analysis. The evaluation process seemed very 
mechanical and automatic, but none of those interviewed' seemed' a strong ^ 
advo.cate of the .system (with the exception of the returning instructor ^ 
debriefing) . . . ' = ' 

' The type. of data collected for evaluation by MMCS is almost 

identical to that collected by the evaluation unit at _*:^FB. 

Methods of analysis are also similar. However, the use and utility 
of the data in the fwo -systems is very different. . . 

At APB, the graduates and their , supervisors are looked 

upon as a very important, perhaps most important, source of information 
for curriculum revision. Consequently; the surveys go out under a, 
directive of" the commandant and over 80% are returned. There is also^ 
a strucutre and effort to involve instructors in the evaluation and t'o 
see tKat evaluation results are considered _in course .revisions. 

At MMCS. the motions of evaluation were similar, but not one seemed 
'to consider the effort or the findings very imporfant. Instructors 
received an annual report based upon graduate surveys, but there was 
no evidence that the report played .a substantial role in course^ revision. 
Communication between the development division ^conductin'g the evaluation 
and the division conducting the tr.aining was lacking. ^ 
/ • - . ■ " ■ ■ 

The evaluation effort at MMCS is carried out with technical 
competence and is collecting potentially useful data, but the structure 
separating the evaluation team from the instructional team limits the 
application of evaluation findings. - ^, ^ 

Perhaps the lesson here is that, regardless of .the technical 
' competence* or design of an evaliiation , if the , audience, for whom the 
•evaluation is intended does no t^ feel that the evaluation is important, 
little use will be made o-f the findings. * v 
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5. REPORT OF SITE' VISIT TO SOCIAL SECURITY ADMINISTRATION CONCERNING A 
' MANAGEMENT BY OBJECTIVES (mO) PROGRAM 

Introduction • - , , * ' 

In reading this report- and the inf ormation ^epor ted. in the Program 
Evaluation Questionnaire for thi\S study, it is important for the reader 
to know- that^ the particular course repor,ted,,i% part of a broader set of 
offerings prcfvided by the Division of Training and Career Development "of 
th^ Social Securi'ty Adminis tratioa, and that in some ways the particular 
course, is not representative or typical "of evaluation procedures employed 
in those programs. In 'particular , evaluation of MBO this far is- more 
tentative and informal and therefore less conclusive, for reasons which 
^ will be described in later sections' of this report. 

^ The. PriD^ram ^ . - ^ ' 

j^** . At the time of the visit the Management by Objec'tives Course (MBO) 

had^ been, presented on six occasions, oyer a two year period to group 
sizes , ^ averaging t\'7enty-^f ive individuals - less than 200 staff in all* 
Typically, the course consumes three ftill days of workshop activity 
usually spaced within a; thirty day period with Consulting provided as • 
, requested thereaifter, \^ . . 

The MBO course is one o'f , an extensive set of "inservice offerings 
develope'd by the Social Security Administration to staff at all levels. 
■iThe^MBO course, in particular, is offered to second 1-evel supervisors 
and above all in Social Security Administration departments, most of the 
in-^iouse courses conducted by the General Skills and Management Training 
Branch are ndt required, but offered as opportunities available * to ^taff . 

Although top level adminis trati'on of the Social Security Adminis tra-^ 
tion has indicated an agency interest in MBO, the program was initiated 
by the Management and Supervisory Section with approval of Division 
' Management," 

Although program goals are pre'sented to participants during the first 
' hour of the wbrkstiop^ and in a flyer beforehand, these were not available 
to the*writer and cannot therefore^ be -related meaningfully to the eva-lua- 
' tion or to this narrative.' In general^ each participant is encouraged to 
develop a limited but , improved set of outputs and results he would like 
to accomplish by certain target dates', after he has learned the jargon 
and'^read wi-dely in the MBO literature. Parenthetically, the writer notes 
that in his opinion, the readings are significant and wide-ranging in 
application.^ , .• ^ • ^ 

The Evaluation ^ . , 

V \ ' . W ' ■ , ■ ■ ' • 

\ ^^At this stage in the development of , and experience with MBO, the • 

evaluation can only be described as informal and primarily observational * • 

t • ■ ' 

The overriding value for the evaluation activity stems from the 

desire to assure the trainers, themselves, that something worthwhile for 

the participants is going on,, rather than from any charge from above to 

document the "impact of the effort. There are several circumstajices which 
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cc..;.tribute to the present form of the evaluation - the new body of ; 
content itself (MBb is a relatively new set of ideas), the varied 
backgrounds and responsibilities of staff taking the course, and 
finally the view (widely shared by many current users of >[B0) , that 
its potential for organizations lies in its ^'process", not in its' 
particular form, . . . ^■ 

Additional comments about, these matters and how they ' presently 
constrain evaluation procedures (at -least in the setting reported- here) 
will be offered in later sections of this report. 

As indicated above, the evaluation is conducted in-hbuse by members 
of the training staff. Although it was not the intent of the visit or 

-this study to evaluate . evaluators , it can be reported that all evaluations 
of courses are prepared within the agency and that no special individuals 

.^ithin the training staff are designated a~s specialists on evaluation. 
Prob'kbly all .have experience, expertise and qualifications I'n their 

backgrounds $v ^f icieht for the conduct of meaningful evaluations. 

\^ 

Evaluation* is conducted with funds provided for by the org^izational 
and di"isional\ budgets . Evaluation data are reported to the nextXhighe^t 
management level, prima^^ily as information for allocation decisions 
regarding the mix of specific training efforts. 

Evaluation Methods v • 

As indicated earlier, methods employed in this particular course 
evaluation were informal and observational. [ . ^'^ 

At the conclusion of the course, a standard course-ending question- 
naire is administered. This three page questionnaire reflects ratings on 
nine specifics of the. course such as; the degree that courses, met 
expectations^ the degree to which the cours-e was pertinent* to needs, 
degree of interest, the effectiveness of the course leaders and the like. 
Another section, of the questionnaire, the respondent_to make statements 
about' their expfectations for the course, their re^ons XNQr^^^^t^ndance, 
and why the course was beneficial, ' In a third sec tien^^ the respondent is 
asked to m^ke suggestions for improving the object;kcns, the^-lormat, the 
schedule, like leaders, the subject matter, the materials of o^s tructidn, 
the facilities y and other matters. ;■ 

Rating scale positions. are not operationally defined so that it is 
difficult to pool the rating-s across groups- and times or to compare ''tbem. • 

Approximately three mdnths -after the conclusion of the workshop, 
participants return for a'critique session. with training staff. These 
sessions offer training staff a further opportunity to submit; suggestions 
<or modification of course content as well as the opportunity for the 
trainers to see and discuss specific work products developed as a result 
of the course. The generation of goals , rejulcs, schedules, ^etc. during 
the workshop was referred to as simulation ^ at the time of the critique 
actual' work products are available. 

For some courses other than MBO, a four-page questionnaire is 
administered as a further 'follow-up evaluation to assist training staff 
to collect feedback for course improvement. 



Given the constraints listed earlier, and cons quently the focus 
on ''feedback" as the evaluation goal for this cour.se, the training 
staff report satisfaction with the informal evaluations. They indicate 
that the informal procedures briefly described thus far have provided 
them with information which was used to modify (improve) the cooarse on 
each occasion it has been offered. 

Finally, informal observation by training faculty is used also as 
a basis for course modification. 

Evaluation Results and Effects 

The experience with evaluation of the MBO course thus far has served 
to illustrate and uncierline several persistent probliams which deserve 
brief descriptions here , • 

First, training is a staff service function not on line. On the one ■ 
hand this is seen as having definite advantages for providing opportunities 
for ' learning. On the other hand, from an evaluation viewpoint the stafi 
function removes the trainers from ongoing operations and limits almost 
entirely opportunities for monitoring or ^'process" evaluation. 

Second, the organization as a whole views MBO only an an aid and 
takes great pains to avoid even the suggestion that MBO should take on a 
particular form or set of specific procedures. It is the desire of the 
management itself and the training, staff to offer MBO asi a set of broad' 
principles, a "process", which can be applied in any way consistent w' th • 
the needs and experience of . various operating departments. Yet, a 
"process" is more difficult to measure and evaluate than a specific form 
or outcome. This general difficulty with the measurement of "process" 
coupled with severely limited opportunities to observe "process" (ongoing 
operations), a^ indicated in the previous paragraph, compounds the problem. 

And finally, the problems of evaluation are further compounded by 
another instance of what is generally referred to as the "criterion ^ 
pro.blem" the lack of common outcomes, in the form of behav^ior, indica- 
tors and work products, which' could be used as a basis for the evaluation. 

In short, the understandable posture the agency takes toward MBO as^.a 
techniq-ue, the limited opportunities for observation by a staff unit off 
line, tfie relative difficulty in measuring a "process" (as distinct from 
knowledge^ products and other more discrete outcomes), and the widely 
varying settings within which MBO is applied, represent major constraints 
for evaluators in this and other settings when those characteris tics come 
together, 

A jnajor effort presently unden^ay by the training staff, which is an 
effort to catalog and roap the interactions between some 500 specific 
occupations and exposure to specific learning opportunities, may provide 
some "further insight into and refinement of evaluation procedures. Staff, 
there are not optimistic, however. 

It remains a-fact that outcomes from learni: opportunities for 
managers remain difficult to document when "those outcomes are expressed 
in "process", "stylistic" and other more global term^- 
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•REPORT OE. SITE VISIT TO U. S. DEPARTMENT OF AGRICULTURE CONCERNING 
A ^SUPERVISORY TRiVINlNG - PHASE I PROGRAM ' . 



The Program 



is an agency of some 9-10,000 



employees within the Department of Agriculture. Originally the agency 
had both a regulatory as well as research function, and until 1969-70 
had a staff almost twice the size of the present force. At present 
the agency has.no regulatory function, and its staff consists largely 
of senior and, junior scientists, research , assistants , and the support 
staff necessary for these positions. 

The Supervisory Training - Phase I is available to Anyone within 
the agency" who supervises other employees, and is a requirement for all 
new supervisors. Many of the employees of 



have scientific backgrounds but little or no inanagement 'training and/or 
experience. The training program is intended to develop management 
abilities in the supervisory personnel as well as to insinuate a ^ 
particular strategy of supervision — -a problem analysis approach. Entry 
to the program is not on a competitive basis, and, with the exception of 
mandating the training for new supervisors, there are noj formal selection 
criteria. The program has been operative since 1970 and; .claims 3000 . 
trainees. 

■ The- Supervisory Training Course - Phase I generally covers two 
40-hour blocks and is given to all newly selected first leyelv supervisors 
either before they assume their duties or as soon thereafter as possible. 
The program is offered at 25 sites and is said to accommodate employees 
from GS5 - GS15. The training groups are generally small, numbering not 
in excess of 204- persons.' The course is most concerned with what its . 

. framers call "Dealing Effectively with People." The course director is , 
seen more as a facilitator than an instructor and the trainess do a 
great deal of interactive work, in small groups. At the conclusion of the 

•course- each participant has an opportunity to complete a rather informal 
evaluation form indicating his feelings about the. training . There is - 
no systematic evaluation of -the individual trainees. 

, The Evaluation 



As revealed by the Program Evaluation Questionnaire received from 
this program, there has been very little formal or even systematic 
evaluation of Supervisory Training - Phase I. Since some' occasional 
attempts to gain "feedback" were made some . response to some of the items 
of 'the site' visit outline seems in order. Items have in some cases been 
combined for convenience and in light of the brevity with which they 
could be covered. 

What kind of evaluation, carried out by whom» who paid for? 

All evaluation of STP~I has been in~house evaluation. All of these 
evaluations have been rathdr informal and conducted by program staff 
The earliest "follow-up" was headed by the staff which devised the 
program. Subsequent follow-ups have involved, in the main, trainee 
■ reaction to the program. Although program administrative staff has 
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done some impressionistic rating of the program, there has been^-^^ 
no systematic assessment of training leaders. As in~house 
evaluation or follow-up, all costs were borne by the agency itself. 

Who were the audiences of' the evaluation? 

The first evaluation follow-up was actually carried out for the / 
program developers. A secondary, implied purpose of the program 
evaluation was to provide some guide or format for potential students 
and supervisory personnel not exposed to actual training. The main 
objective of this effort was to help program developers "refine" 
their course. 



What were the expected outcomes and benefits of the evaluation? 

The original follow-up was stated to be an attempt to determine to 
what extent those who had " completed the training were applying its 
concepts in their work settings. Presumedly, if the concepts were 
not taking hold, the training package would be rethought and possibly 
modified, if the concepts had influenced- supervisory behavior 
significantly training would be termed successful. 

What did the evaluation (s) comprise of? 

The first evaluation or follow-up approximated formal, systematic 
evaluation more closely than did later feedback gathering efforts. 
In this first effort, which took place shortly after the course was 
devised, two pilot groups of 6-12 persons were run through sample 
course segments with the- designer of the material's serving as co'urse 
director. Certain revisions were made in the course, based on • 
feedback from these pilot groups. In addition to the pilot groups, 
one field site was selected- for follow-up six months after' the 
training. As was stated .earlier , the purpose of the follow-up was 
to determine the extent to which supervisors were* making use of 
the concepts of their training in their actual job settings. 
Seemingly, individuals wp.re asked to cite examples of use of the 
concepts of the training, and a collection of "mini-case studies" 
of success was compiled. The conclusion reached was that the 
course was successful' and would continue to be so unless there was 
"a substantial change in the 'type' of personnel becoming supervisors 

Was the evalu^ation carried out with technical competence? 

There seems to be considerable doubt as to whether the term technical 
should be used to describe the evaluation of this program.- Although 
those who conducted the training are probably ^^competent trainers, 
there is no. evidence of staff evaluation expertise. 

Did the evaluation accomplish the' evaluation goals? - 

According to the program director , the evaluation accomplished its 
goals. The program director readily admits that this is an 
impressionistic assessment. 
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Did the evaluation seem to help the audience for whom it -was 
intended? Were any changes made as ^ result of the e valuation? 

The ans\7er here seems to be yes on both coui.ts. The program 
director insists that the feedback obtained from the follow-up 
helped immensely and that a number of modifications were made in 
the program as a result. 

Did the evaluation look for unintended outcomes? 

Not in any systematic way. This is true since the evaluation wasn't 
systematic to begin with. However, since trainees gave open-ended, 
narrative feedback there must have been some mention of side-effects. 

What was good about the evaluation? 

The program director contends that the program modifications which 
occurred as- a result of the evaluation made it most worthwhile. 

What were the problems? 

From. the program director's perspective there were no problems 
with the evaluation. It seemed unfortunate that a program that would 
include such detailed training material would give such scant 
attention to evaluation. 
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REPORT ON SITE VISIT TO FORESTRY SERVICES CONCERNING A PROGRAM ON 
CLEAR WRITING 

and Mr, are training • specialists in 

the Training Section of the Division of Personnel for Forestry Services 
USDA in the southeastern region. The Training Section of the Division 
of '-Personnel is responsible for techincal, management, and communications . 
training for the 3000+ Forestry Services employees in the southeastern 
states from Virginia to Louisiana. Training ranges from courses in the - 
''Use of . Pover Saws" to "Management by Objectives" and "Assertiveness 
for Female' Workers." Although most of the technical training is 'conducted 
by independent or government agencies under contract to the Division of 

Personnel, Mr. and Mr. personally conduct some 

training at the regional level In management and communications skills. 

Courses are offered at both field and regional levels for groups 
ranging from a minimum of 20 participants to an average of 30-60. 
Participants may be required by their supervisors to attend a particular 
training course, or they may voluntarily participate. Each employee is 
required to undertake a minimum of 40 hours of training each year. 
Students are not usually evaluated directly; their participation in \ 
training is noted in personnel records. 

Ninety days following any training course, a sample of 10% of the 
participants are mailed a follcw-up questionnaire by the Training Section 
of the Division of Personnel, ^e supervisor of each of these 
participants is, at the same time, sent a parallel form of the evaluation 
questionnaire. The same questionnaires are u?ed for all courses, and are 
consequently very 'general and use au open-ended response format. A 
return rate of 50-^75% is typical. 

When ah "adequate" number of questionnaires are accumulated, the 
training specialist "reads 'through them, noting patterns of responses 
and possible revisions for future training . . The 'questionnaires are then 
routed to the person or agency Sitectly- responsible for requesting or 
conducting the training^. " ^ . ' 

Evaluation of training is required by Civil Service regulations, 
but no guidelines are given for the type, or extent, of evaluation. Some 
limited influence ^and guidelines are exerted by the National Office of 
Forestry Services, but, for the most part, the regional offices are 
autonomous and the Training Section determines the quantity and type 
of evaluation activities. 

There is no separate budget for evaluation activities. Evaluation ; 
is an adjunct to training. Some 1.2 million dollars is spent annually 
by the Forestry Service for training activities. 

The principal audience for the evaluation is regional management 
"who want , some- assurance that the dollars being spe;^t for training are not 
- being wasted.- A secondary audience is the "instructional agency, for 
the improvement of .course content. There are few technical aspects to 
this evaluation. The 10% sample is selected by a clerk given no 
directions for how the sample is to be. selected. There is no 
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quantitative analysis of data, and, in many cases., no formal attempt 
to draw conclusions from the data other than the reading of responses 
by the training specialist. 

The evaluation efforts have resulted in some information of .the 
type sought by program managers. A recent trainee of the Management 
by Objectives course reports a cost saving of $50,000 in one year as a 
result of the training. For the present, the identification of such 
outstanding case studies is a goal and accomplishment of the evaluation. 

Obviously, this evaluation effort is limited in design, analysis, 
and purpose. The sampling method probably does not provide an accurate 
picture of the effect of training for t.he entire population. The 
analysis allows evaluator and management to select those results that 
favor a foredrawn conclusion, and this is, intact, one of the purposes = 
of the evaluation. ^ 

There is some indication that the evaluation efforts of the 
Forestry Services' Personiiel Division are continuously improving. There 
is planning for a system of need assessment to determine training 
priorities and there is an ^ effort to have training and evaluation of 
training become a required part of each supervisor's annual management 
plan. ^ 
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REPORT OF SITE- VISIT TO JG INSPECTION SERVICE, U.S. DEPARTMENT OF 
AGRICULTURE, 'CONCERNING AN EXECUTIVE DEVELOPMENT « PROGRAM 



Program 

JG Inspection Service is a regulatory agency in the Department of • 
Agriculture. It has approximately 16,000 employees. Because of the 
scientific nature of its mission most managers have a science or 
veterinary medicine background with little man^^ement training or 
experience. The Executive Development Program is intended to provide 
the basics of management principles to individuals entering the. agency 
at relatively high GS levels (13, 14 & 15) because of their scientific 
training but with low management or supervisory experience. Candidates 
volunteer for the progr^ and are aslected on the basis of a number of 
criteria including their GS rating and ratings of supervisors at least 
two levels above them on "potential to accept greater, responsibility. 
In the first cycle which is still underway, there were, about 450 
applications of which 120 were selected for tlu-^ program. 

The program extends over a period of about two year-o with a total 
commitment of time of fifty-eight days. There are eight courses involved 
in three phases with an elaborate individual evaluation system. The 
evaluations are for the students information and do not go into any 
official file or^record. Students may drop out of the pir-ogram but no 
one is failed or asked to leave. 113 of the initial group are still 
participating as they go into the third phase. Following course two, 
which is called Self-Awareness ("a modification of transactional analysis), 
the participants complete a lengthy self-evaluation. As each course is 
.completed, the faculty or counselors or supervisors complete a similar 
evaluation for each participant. Copies of these evaluations are given 
to the student as a progress report. It is planned to develop a 
certificate of completion of the program which will be awarded to those 
who finish. 

Evaluation . . 

For such a carefully planned and documented program with the great 
deal of emphasis on evaluation , feedback to participants there is 
surprisingly little planning or execution of a program evaluation,^ However, 
to be responsive to the charge, the questions listed on the Site Visit • * 
Outline follow with., answers that are of ten very brief because of the 
nature of the evaluation. . ' 

Sections 2 and 3 - What kind of evaluation? Who carried it out? 



An organization titled 



Associates, Inc. apparently has a 



contract with the Department ■ to provide consultative services in the area 
of management training. As part of that arrangement they cover the 
JG program. So far their evaluation seems to be one of trying to determine 
if the various. agencies within USDA are in compliance with 'the regulations 
and guidelines directing them to haVe an executive development program. 

Thus,^ the outline of a briefing for" USDA Management Council by ' : 

Associates, Inc. shows an evaluation of the progress the USDA agencies 
are making toward being in full compliance. Apparently this evaluation . 
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was done, by reviewing planning documents, interviewing those responsible 
for implementation and examining courses, procedures, manuals, etc. 
used by the various agencies. 

■ While • Associates, Ind. proceeds with its compliance 

evaluation of all USDA agencies an individual consultant (non government) 
is serving as an evaluator of the specific JG Executive Development 
Program. The consultant was described as a psychologist who had worked 
for a management consulting firm and for the Federal Government's Office 
of Investigations. It isn't clear just exactly. how he was selected 
but it seems that those "concerned sort of just knew him and he whs hired. 
He reports to the administrator of JG.|^ His evaluation seems to be 
subjectively based on interviews with participants, faculty and deputies 
who do the indivrdujil evaluations and on the basis of observations of 
sessions in progress. His reports were not available but apparently 
he has passed on suggestions and made' recommendations orally to the 
director. He is supposed to be developing or has developed a 
questionnaire for the participants. 

When asked about an evaluation of the impact of the program on the 
participants', careers, for instance comparing^.par ticipants to non 
participants, the director of the program indicated that he himself sort, 
of planned to take care" of that. He also explained that it would be 
difficult to conclude much about the effectiveness of the program that 
way because people might have been promoted any;;ay. 

Section 4 — Who paid for the evaluation? 

The Federal Government pays for the evaluation through contractual 
arrangements with the USDA. - 

Section 5 - VJho were the audiences of the evaluation? 

The -audience of the individual evaluation was the participant group 
undergoing the training. The _ Associates, Inc. work was aimed 

,at high level USDA office with responsibility to comply with executive 
'orders. The individual consultant work seemed intended to provide 
\feedback directly to the program designers and administrators. 

Sec tiojg 6' - What were the expected outcomes and benefits of the evaluation? 

The ^ Associates, Inc. effort would seem aimed at producing 

results that vould allow the 'department to state that it is in compliance 
or if not. provide information to direct efforts to. be so. 



The individual i^qnsul tan t outcome seems aimed^ at more of a 
monitoring kind of goaixthat allows for quick adjustment in the program. 



Section 7 - What did the evalAaation comprise of? 

There seems little that can^ke added here not ^already covered under 
section 2. The conclusions are not^vailable at this time and may-r,never 
be, given the kinds of things being dolae. 



S ection 8 - Was the- evaluation carried out with technical competence? 

Given the non scientific approach it doesn't seem possible to-answer 

this in the sense it ,is asked. As far as can be told, the people doing 
the„ work have competence. , 

Sectio-n 9 - Did the evaluation accomplish the evaluation goals? 

This can't really be answered because the first cycle isn't complete 
at. this time. However, as far as the Hireetor of the program is concerned 
he is satisfied with the individual consultant ' s work. Presumably, 

Associates, Inc. will at some point report that the USDA is^ 

in compliance which is the goal. 

Section 10 Did the evaluation seem to help? Were changes made. ? • 

The answer to both "these questions would have to be yes. The 

Associates, Inc. reports help implement the program by alerting USDA that 
certain things are or are not happening which results in changes. "^^^^^ 
individual consultant's feedback has resulted in some modifications inl 
the program. However , the bigger question of the value of the program 
doesn't seem, tobe being a<ddressed. . . . 

Section 11 - Did the evaluation look of unintended outcomes? 

The answer, here, would have to be: not systematically because the-, 
evaluation itself is not systematized. On- the otherhand the individual 
evaluations of the participants allow for such a wide ranging coverage . 
that the answer would be--yes. 

Section 12 - What was .good about the evaluation?. . 

The ongoing' professional judgment by the individual consultant 
allowing for p>rompt adjustment in the program seems worthwhile. 

Section 13 - What were the problems? « - 

><? 

It was disappointing to see the elaborate care in planning and 
developing the program including the individual evaluation of. participants 
with no similar .effort made to determine effectiveness or impact. The 
call for ExecutiiC^e* development Programs was issued as an edict. They 
are *'good" per se. ^ Why would ■ anyone go any further and gather data that 
might be analyzed to conclude that the whole thing is a waste and managers 
come into being -by a political pi;^ocess rather than a merit procedure? 

Section 14 - .A±e there special lessons? 

The use of high level managers in some of the participant evaluations 
seemed a particularly valuable device in that it makes the training 
program more a part of the whole agency -rather than a sort of school 
separated from- the reaiities"^ of the ongoing operation. - The old problem, 
was there of not thinking through ways of evaluating the effectiveness 
or impact of .a program resulting in a little fuzziness in answers to 
questions about how it' will .be known if~ the whole thing did any good. 
Of course ,; transactional analysis is good for.- better managers. How 
could anyone question such a thing? 
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REPORT OP SITE VISIT TO . INN UNIVERSITY CONCERNING A FOOD 

AND BEVERAGE >IANAGEMENT PROGRAI-l 

The agency administering the program ^nd evaluation was the 



Inn University ( lU) , a corporate supported institution whose primary 
goal, is the upgrlding of _______ 1?,^ staff with the ultimate 

expected outcome revealing greater efficiency in overall operations and 
thus an improved profitability picture. 

The programvbeing administered at the time of interview was entitled 
Food and Beverage .M<magement (FBM) . It was designed to provide a 
thorough exposure for^all phases of the food and liquor operation of 

l^ns via a variety of instructional approaches. The class 
size attending the current program consisted of 35 participants from 
almost as many states. The Food and Beverage Management Program was 

designed and developed by staff personnel at the . Inn University 

, Mississippi. The programs offered by _IU vary in 
length and extend, from one week to four weeks, contingent on the type 
and design of the program. The ultimate goal of each program is to help 
develop and extend the * "expertise" of each program participant to insure- 
a greater efficiency of operation on their own site. . . 

Students are evaluated by a variety of measurement techniques-. 
These include paper and pencil tests, performance tasks, and an. ongoing 
subjective evaluation by the team of instructors. 

The evaluation of the participants is initiated through the.^use of 
a paper and p-encil pretest. This instrument is administered to all _IU 
-participants prior to their formal classroom instruction and then is 
readministered'^at the completion of the instructional program. The 
^program -eyaluators are primarily interested in raw ^score gain as revealed 
by 'the pretest-posttest administration and considerVthe participant and 
program a success if a net gain between the' two tests is-noted. The goals 
of the^ evaluation include a comprehensive review of course content for 
■applicability in the field, a profile of individual and group growth, 
and a determination of the relationship of the instructional program to 
the objectives of the Food and Beverage Management Program. . 
^' , ■ ■ ^ 

The evaluation of the Food and Beverage, Management Program 
^participants was basically carried out by the classroom instructors. 
The instructors were selected because of their expertise, experience, 
interest, and ability to communicate quite easily and effectively with 
others in both formal and informal instructional settings. 

Fo-r this" paifticular program the instructor s/evalua tor s selected were 
all lU staff members and represented in-house personnel. 

Evaluation is an integral part' of each program design and as such 
is financed in the same way the "Univ'ersity^' is financed. As part o'f 
--the corporate policy as defined by the International Assaciation of , 
Inns, 'each Inn is assessed one cent per room per day to 
"Maintain the^ "University." This revenue, which is yielded by both 
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corporation owned and franchise owned Inns is sufficient to ma^tain 
the ..present staff development and training programs. In addition to 
the assessed revenue, each participant in a ^lU program pays for room 
and board while in attendance. . - 

The evaluation results for each program are Summarized and shared 
with members of the corporate board. This information is also presented 
in appropriate corporation publications for circulation to all Inn-keepers. 
In addition, individual evaluation inf orma^?.on is provided to all 
appropriate franchise owners who sent a ^participant (s)'^ to--- attend the 
_IU program. Participants of each of the programs are also continuously 
provided information regarding their class performance and achievement. 

To the corporation, the expected outcomes for -having staff and - 
franchise o\^ers attend voluntary or mandatory _IU .programs 'are directly 
related toS:he improved efficiency of Inn operation. .The corporation 
has assumed the responsibility of personnel training and through their 
lU program supports the franchise owner by designing and implementing 
"staff development programs. The program developers and staff benefit 
directly from the evaluation by reviewing individual and group reports, 
and interpreting the results in light of the program objectives and goals. 

The evaluation of the Food and Beverage ^lanagement Program consisted 
of a pre and posttest (same form) administered , to all participants and 
unit tests designed by the teaching staff referred to as criterion 
examinations (CE) . The objectives for the program were behaviorally 
stated based on the approach used by Mager and the unit tests were 
totally based on course' content. In addition to the paper and pencil - 
tasks, the participants were also evaluated on the degree of skill displayed 
in the use of a variety of business machines they were taught to operate. 
Rounding out the participant evaluation process was the^ completion by 
each instructional staff member of a Participant Evaluation form dealing 

ith affective" evaluation. On this form eight categories are defined 
with a scale range of from 1-5 for each category. 

The overall design for the evaluation program was developed through 
tha utilization of a team approach. Instructional staff members were 
responsible for the development of the pre and posttests and the unit 

tests.. Both Mr. .. and Mr . of the _IU's Educational 

Department were involved at a review level for all measurs developed and 

used, ^i^'' ^ developed tha^ Par ticipant Evaluation form 

specifically for the _IU programs." All measures developed were used 
without benefit of field testing and limited analysis of both individual 
and group data was evident. Provisions f9r both individual and"summative 
item analyses have just been arranged and presumably should be quite 
beneficial when *used with future programs. 

^ \ 

The design ^of the Food and Beverage I'lanagement Program incorporated 
seven inter-related units' to be- covered by the instructional staff. 
These included: Employee-Employer Relations; Cost and Controls; . 
Advertising and Promotions; Restatirant Operations; Banquest and Meeting-. 
Rooms; Bar Operations and Standards; Goals and Awards. ^Each unit 
presented had a criterion examination (CE) developed specifically for^ 
its course content and each unit test was reviewed completely with the 
participants -following the administration and. scoring. ^ ' . "C 
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Instructional staff members as well as central office supervisory 
personnel believe their evaluation design to be an effective one. 
Their expressions are based on the evaluation program satisfactorily 
meeting the evaluation goals of expanding and refining necessary 
management level hotel skills so that a greater efficiency of Inn 
operation will be obviously' notable. Hotel procedures have been 
carefully studied by._IU supervisory staff so that course content arid 
evaluation measures reflect current practices. The, director said of 
the evaluation: '\..we were able to draw some conclusiorts from the 
overall evaluation which should provide direction for an action plan." 

The evaluation pointed out quite- significantly the wide range of 
experience and expertise that the participants brought to the _IU program. 
In terms of course content this ^might suggest an individualized approach 
'for some^^ participants and also suggests the possibility of grouping for 
instructional purposes. Both of these alternatives are being * considered 
for possible introduction. in future- courses. The evaluation program 
for. participants helped to identify on a daily and unit basis individual 
-strengths and weakiiesses, and for instructional staff indicated areas 
of the ins tructional .program requiring review and possible revision. 

There was no .intent on the part of the evaluation program to look 
for anything but program outcomes originally indicated by the evaluation 
design; however, interaction meetings between participants and 
instrucational staff "were responsible for the earlier .reduction of* the 
Food and Beverage Management Program^, to a four week period of time 
rather than the previously established five week program. ' " 

. The primary function of the evaluation design was. to help assess' 
how effective the Food* and Beverage Management Program was in better 
preparing the par ticipalits to operate* with .greater efficiency the 
"Inns" entrusted to their management and supervision. The evaluation 
appeared to. measure the attainment of certain necessary skills. and the 
acquisition of certain pools of informatiqh as predetermined by the 
education department with the assistance of the instructional staff, ^ 

Both ^participants in the interview raised questions about the / 
design of the paper and pencil measures .developed for use as unit tests 
by the instructional staff. Some additional questions were raised at . 
this time, regarding the reliability of the instruments used but further 
questioning revealed no futur.e action was being contemplated to improve 
the reliability picture. 

Several options are open fot consideration in terms of 'developing 
higher, reliability of test measures One suggestion would be to enlist 
the aid of outside experts to assist in the building of the unit 
examinations. Field testing of.^the measures including an item analysis 
should be. undertaken by the _IU staff to determine statistically what ^ 
items should be included in the-, criterion test measures. Again, 
outside expertise should be utilized, if .deemed -necessary. 
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If >the evaluation program were to be physically, extended to 
include an, on-site visitation of selected programs participants 
fox the purpose af determining the effective application of the • 
-program content , .th/ feedback might realistically support _IU findings ^ 
'or provide ^Ilf ins tructional staff with insights ^r program modification.; 

If one may take the liberty of generalizing from an interview it 
would appear j udicious ' f or organizations developing evaluation programs 
to seek outside professional assistance if the talent required doe? not • 
reside ojc is not available from within the organization. This action . 
will serve two functions: that of -providing a professional quality to 
•the overall. evaluation program and secandly, to ^allow for an interchange 
of ideas, values, etc. between intetnal and external professionals,- 
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REPORT OF SITE VISIT TO' XY£ ' CONCERNING A SALES FUNDAflENTALS PROGRAM ' 



The Program - - ^ / - 

The Sales .Fundamentals -^'Program (the course, I was informed, is 
actually referred to as "Effective Presentation of Ideas") is one of 
siore thaa 30 recurring training and^ development programs offered by 
XYZ*s Employee Development Department, which is a department within the 
Division of Personnel and Industrial Relations.' The Department Is 
responsible for providing development and training assistance to all . 
departme'.^al units 'within XYZ. -The three "staff members of the Department 
either ccuauct training sessions themselves or else train instructors at 
various" locations to conduct programs they have prepared. 

The Sales-Fundamentals Program, like all other training programs 
^.-provided by the Employee Development Department, is conducted by request 
/qrxlyy Contrary ro what the course name may suggest, the program is not 

just for new sales personnel. Mr. indicated that course 

participant^^a^ie^^ sometimes employees of ^-companies that have. 

- beelTtxe'^ly acquired by XYZ. -Course participants .may_jiave jia^_x 
extensive^ sales eKperience^X-Si^me ur^^ . . 

Courses are requested by executives in. ether departments or divisions 

for personnel whom they supervise. Mr> * 1 said that requests 

are made on the 'basis of personal relationships. That is, executives^ 

request "courses for their employees simply because they know Mr. % 

and are' familiar with the services he offers.' Iri the words of Mr^?^^^ 



CLilVJl J, CAUl*..^-L. JUCi i. ¥T-i-k»l.l. wfct.*- ^ — ■ 

it's^a ''you scratch my back and I'll scratch yours" kind or arrangement. 



The Sales Fundamentals Program has been- offered since 1960. 



Mr, 



has beeti in his present positi^on for two years. During his tenure the 
course has been updated (part of it is brand new)., ^H.e estiiD^tes that 
1000 trainees have participated in the course since its inception. 

^ The course consists of 16 hours of in-class time usually divided into 

t eight hour segments or sometimes 4 four hour parts. . Mr. 2 

showed me' the objectives of the program, which are presented to students 
as part of a slide presentation. (The course relies heavily on a multi- 
media approach. ) The objectives are: . ^ 

' 1) To provide each participant with a perfect standard for selling. 

2) To develop a vocabulary for selling. - 

3) . To acquire a knowledge of the basics and fundamentals of selling. 
* . 4) Tq give a new' insight tp true product knowledge. ^ 

5). To be able to teach others. 
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Mr, . realizes that these objectives are broad and general 

and, he thought, uoncontroversial ("motherhood and apple pie"), / 



The Evaluation 
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The evaluation that, is carried out consists of --^student, participants 
.completing a Program, Evaluation Form at the end of the course. The 

idea for using the form was Mr, ^There was no dvi^rective 

for evaluation issued by^fl superyisor, 

Mr. • . sees the evaluation in"" a number of interesting ways. 

He stated quite frankly that one of the major purposes was to justify 
his yearly raise and "to show my boss (the Vice President for Personnel 

and Industrial Relations) how .good the program is." 

showed me tabulations based on a number of program evaluation forms. 

.. . The results are, in general, very positive suggesting the . 

objective has been accomplished. . ; 

In- llne-^with" justifying his y^ Mr. . showed me 

his "ego file," which contains supportive letters from course participants 
. or from their supervisors. indicated that letters of this 

\ nature are -quite frequent. That is, about four of every five times they 

conduct a course, he gets a letter — usually from the executive who 
requested the course. . ii ' 

Another purpose in conducting the evaluation is to get feedback on . 
the performance of the people that he supervises* — i.e., their ef f ectivenes-s 

in presenting the programs. In this regard, feels that he has 

succeeded -in "getting an overall feel for depar,tmental effectiveness," On 
the other hand, he suggested that the Vice President . (to whom he provides 
monthly summaries of his evaluation results) "doesn't even look at them." 

Mr. / seemed to think that the evaluations were 'Somewhat 

useful to him and his three-man staff in making modifications of courses. 
Suggestions from course participants have also been used as a basis .for 
the development of new programs such as the Advanced :S aids Program. 

From participants comments has also learned 'some 

things which he considered toVbe interesting. For example, from comments 
such as "Why don't. you give our.bo$,ses the same training?" he has realized 
that his efforts have been directed almost . exclusively at lower and middle 
management. As yet, however, upper level management, personnel have still 
not' participated in courses. 

In talking about problems ' associated with the* evaluation , . 

mentioned that he had been very successful* in getting participants tq return 
the evaluation forms. He estimated that/"99%" of them return the forms. 
He wa&, however, somewhat disappointed in that he felt that his efforts may 
have degenerated into more of a popularity contest than ^n evaluation. With 

regard to the comments that, partiqipants make, mentioned that the 

comments made -by- first-time participants are usually less valuable than those 

made by employees who have participated in previous programs. _^ . 

speculated that the reason might .be that first-lime participants are not 
aware that they are expected to make suggestions^ at the end of- the course. 
He thought that he might mention this to participants at the beginning of 

future courses. T^ no ' 
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It was clear in talking to Mr. that his evaluation • 

efforts are not directed at assessing the performance of course participants. 
Ir. fact, student evaluation is -something that avoids ^'rocking, 
the boat.'* He did mention that he is sometimes asked by a course 
^.participant's supervi'sor to make subjective comments about the par ticipant's 
performance in the course. It appears, however, that these requests are 
infrequent. ' ■ * . . 

' When we talked about outcbmes of the program, mentioned 

that it would be possible to formulate more objective criteria for program 
success, e.g., increase in sales volume. However, he was' extremely 
reluctant to do so, indicating that he might be ''cutting his own throat" 
if the evaluation showed no significant impact on sales. 

With regard to establishing' performance criteria, . showed 

an awareness of some of the problems in using such criteria. For example, 
he mentioned that a number of other variables (the state of the economy, 
e.g.,) would make a criterion such as increase in' sales less useful because 
of their effects. • . . 

Regarding . qualifications, he has had 14 years of 

industrial training experience, and has a master's degree in guidance. 
The two trainers whom he supervises have each had five years of training 
experience as well as master's degrees in education and industrial 
psychology. ^ . • 

In summary, the evaluation focuses on improving courses and on 
•assessing t-he performance of the department that offers these courses, 
not on th^^'students that participate in them. Notable was the fact that 
the evai^/ation was self-initiated. Probably the most interesting thing 
about this evaluation effort, however, is the motivation of the evaluator 
and his acute awareness of the implications that any comprehensive 
evaluation might have. 
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11. REPORT OF SITE VISIT TO ABC CORPORATION OF _ 
MANAGING THE XWIJFACTURING OPERATION PROGRAM 



, OHIO CONCERNING 



Section 1 - What is the program like? 

The program, Managing the Manufacturing Operation, is one of five 
units developed by John , Manager of Education and Training for 

ABC Corporation of .j ■ , Ohio. Each of the five units will be 

[presented once during ,each quarterly interval and since each unit, is 
autonomous they need not be-attended in a prescribed sequence.' Four of 
the units run three days and the fifth unit runs four days. 

The program itself was created for use with Division General 
Managers, plant managers, and departm.ent heads. These individuals were 
selected from a national base of company volunteers 'and recommendations. 

A steering committee comprising four group vice-presidents, four 

_g.enjaral managers,, an iadivldual.. f rpm J:he: research .center ,^^_an_d the _ 

assistant to'the comptroller, was established and functioning from the 
outset of the program's initial development . stages . 

The primary goal of the program was to provide a general background 
of information required of management level individuals who at some 
future time woirld be considered, for higher management level positions. 

No formal participant evaluation program was developed, however, 
the participants were requested to evaluate t^e program unitCs) they 
attended.-, ' . ■ 

Sectio n 2 -What kind of evaluation (s) were carried out? 

To evaluate thelprogram. Managing the Manufacturing Operation, a 
program ^evaluation fdrm was employed. In addition to this form spanning 
the overall program, Ispecific presentations were critrqtied following 
their completion. Nd attempt to statistically summarize findings has 
been initiated nor is one planned. 
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Section 3 - Who carried It out? 



The program eval 



uatipn was carried out by John 



who was 



self-selected by virtiie'' of his present organizational responsibility. 
He has been employed ty ABC for six years, is a college graduate, and 
has extensive experience ir^ plant/employee relations. The program design 
and evaluation was an in-hquse function, 

Section 4 - Who paid jor thie evaluation? 



The design, admirlistration, and other related costs were paid for 
through the use of corporate funds allocated by b\dget items for this 
purpose. Plants or otiher irlstallations who sent- participants were 
required to cover all travel and subsistence, expenses for those 
individuals , . * j 

\12G 

1d-40 



Section 5 - Who were the audiences of the evaluation? 

The primary audience for the evaluation was the manager of Education 
and Training. His reactions to the overall evaluation were provided 
to the steering cominittee. upon completion of the program. The manager 
of Education and Training also provided, upon completion of the program, 
informal evaluations, comments', etc. to supervisors of some selected 
participants. 

Section 6 - What were the expected outcomes and benefits of >>the evaluation? 

Expected outcomes were identified as expanding the participants' 
basic conceptual development regarding general management responsibilities 
and the creation and identification of a human resource pool from which 
individuals could be selected for promotion within the corporate 
structure. 

Section 7 - What did the evaluation (s) comprise^ of ? 

• The evaluation of the program consisted of the completion of one 

paper and pencil questionnaire consisting of nine items -by all 
participants and Informal discussions with participants following a number , 
of class presentations\ Consensus of participants' opinions, as shared., 
would seem to indicate that the overall program was viewed as a positive 
experience. " - 

Section 8 - Was the evaluation carried out with technical competence? 
If not, what went wrong? ' 

The evaluation of the program did not ^ include a statistical • analysis 
of data but rather, relied heavily upon the participants' objectiye 
responses to the evaluation questionnaire and to the discussions and 
expressions, following the class presentations. 

Section 9 - Did the evaluation (s) accomplish the evaluation goals? 

The feeling expressed by John . was . tliat based^ on questionnaire 

responses an^^ participant reactions,- the evaluation goals developed>f or 
the program were realized. As perceived by John , all 

participants benefited to some degree ^iby Involvement in the program. 

Section 10 - Did the evaluation seem to help the audience for whom It 
was intended? Were any changes made as a result of. the evaluation? 

The program evaluation appeared to be acceptable to both, the 
steering committee and the education director. As a direct result of 
tne interactive classroom evaluation, one classroom presentation 
directly concerned with financial reporting was considerably modified 
to more^ adequately mee^ the needs of the participants. No other program 
changes were noted as a result of the evaluation. 
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Section 11 - Did the evaluation look for unintended outcomes (program 
side effects)? 

Surfacing from the interactive classroom evaluation was the 
participant perception of the corporate structure being much too 
centralized. Participants expressed a strong desire to be involved 
in policy making decisions, operational decisions, etc., that would 
ultimately effect their installation. While the evaluation design d'id 
ncri originally incorporate identifying unintended outcomes, they were 
in fact noted and transmitted to the steering committee. 

Section 12 - "l^at was good about the evaluation (s) ? Was there anything 
quite -noteworthy? 

John reported that the participants' responses as noted 

in the evaluation questionnaire, were candid and directly on target. 
Their, suggestions regarding instructional presentations were influential 
in the modification of specific presentations for future consideration 
and use. The evaluation .helped to establish credibility by effecting 
"program changes as a direct result 'of the participants' feedback. 

The evaluation did indicate a further need, for additional measurement 
instr-uments to be developed and incorporated into the overall program 
evaluation design. 

Section 13 - What were problems with the evaluation (s) ? How could ^t^Y 
have been overcome? 

The evaluation- questionnaire which was the only paper and pencil 
source dociiment was primarily designed to record participant perceptions 
regarding the areas of program administration, facilities, etc., and 
did not focus pn the substantive portions of the program. It was 
difficult to ascertain how productive the instructional sessions were 
because of the non-use of additional measurement instruments. 

Section 14 - Are there special lessons to be learned from studying this 
evaluation? ' . ; 

Following the conclusion of the program, and the collection and 
analysis of .the evaluation questionnaires, John . noted several ways 
to improve his evaluation process:, 

. . .Future program planning must incorporate to a much greater degree 
the definition of program parameters by cbjectives rather than by 
de-scriptive rhetoric. • """^ 

. . .Input from the field must be more represcil;.^dtive, so that program 
content can be reflective of participants^ oerceived needs. 

In concluding, John commented that much more lead time, was 

required for the planning of a program that was available to him, and 
believes the overall effectiveness of the program/ evaluation would have 
been much more positive with greater lead time. 
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report of site visit to ■ . community college, 
concerning' a medical laboratory technician program 



, FLORIDA 



I met with the Dean. He was -interested in our study and considers 
evaluation an important activity in his college. I also spent a 
considerable amount of time with Dr. , who is chairman of the 

Health Related Programs. 

Program 

This is a two year program leading to an associate in science degree. 
Goals (objectives) are didactic and clinical. Students are evaluated by 
tests, performance examinations (lab techniques), an,d simulations (in 
work settings — local hospitals). 

Evaluation • 

A self-study was conducted by an eleven-person cotmnittee over a 
thre.e month period. Committee included insiders and outsiders. 

Instruments 

An evaluation form was used by the persons who supervised the students" 
in/ the field. It looked like a good ^evaluation form. 

Miscellaneous ■ * " ; ^ 

They seem satisfied with their evaluation activities. They intend 
to include students and consumers of health services on .the evaluation 
cotmnittee in the future. 

The program only began in 1969 for the ' Florida area. 

There are about fifteen to twenty students per year. Graduates take 
the MLT examinations for licensure and the National Registry ■ Examination 
which allows use of the initials MLT (ASCP) after the name. ASCP is 
American Society of Clinical Pathologists-. 

The evaluation was carried out to prepare for accreditation. So, 
we should consider it shared"with us in confidence. 

The Florida Community Colleges are making a strong effort at 
identifying behavioral objectives based- on job analyses in the field. 

Feedback from job supervisors is considered very important and is 
used in evaluating and changing the MLT program. 

Overall Goals ^ - • 

. . -To provide a self-contained two-year training program which will 
bring the student to a realistic * level of training to meet the 
need-5 of employers in the " , Florida area. 

. . .To make the student adaptable to change and aware of his limitations. 
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Conclusion 



The committee concluded that graduates are adequately trained. 
All graduates have obtained rewarding jobs. Math preparation of 
students is weak and not really practical for their work. Math 
requirement has been broadened as a result to include a course in 
chemical calculations as an option for college algebra. Nedd for more 
clinical training has been identified, 'Clinical practicum is being 
lengthened to a full year as a result. 



D-44 



130 



REPORT OF SITE VISIT TO . COUNTY AREA VOCATIONAL EDUCATION CENTER, 

■ ^ KENTUCKY CONCERNING. AN ENVIRONMENTAL AIDE PROGRAM 



General- Description ^ » 

The Environmental^ Aide program is a one-year vocational training pro- 
gram which is offered at the . County Area Vocational Education 

Center in __, Kentucky. It was developed to train assistants or 

aides to work under the supervision of professional, scientific personnel 
in -environmental control type occupations ..• The program, operational since. 
May 1971, is accredited by the Southern Association for Accreditation of 

Schools^ • 

/' 

Funded under the Manpower Development and Training Act,- the program 
provides both^ theoretical and practical experiences which prepare trainees 
for employment in the following occupational areas : soil conservation, 
public health, sewage treatment, water treatment, solid waste disposal, and 
air pollution control. - - 

Training is divided into two phases. During phase one., students receive 
classroom -training covering skills and knowledges required to do routine work 
as required by .the affiliating agencies. This phase includes classroom lec- 
tures, laboratory procedures, field sampling, and analysis. 

Trainees, who have successfully completed phase one are placed with 
affiliating agencies who accept them for supervised work experience on a 
rotation basis. During phase tV7o, trainees spend four days a week with the 
agenpies involved and one day^,back in the classroom for reviewing work expe- 
riences, recording data and information, and planning. The instructor makes 
periodic on-site visits to the af filiating . agencies in order "to coordinate 
this phase of training and to check on trainees' progress. 

Entry into the program is initiated by recommendations from local 
Economic Security agencies in Vocational Regions 1-5 who screen prospective 
trainees to assure that they have acceptable GATE scores in accordance Vith 
established occupational cri'teria.' The program coordinator and instructor 
interview prospective trainees before formally admitting them to the program. 

* / - 

There is only one class in the' program and it has a regular enrollment 
of 18-20 trainees. The class meets weekdays from 8:00 a.m. to 4;00 p.m., 
40 hours per w^^ek, and 50 weeks per year. Altogether the program provides 
the trainee with 2,000 training hours — 1,200 in phase one and 800 in phase 
two . . • 

Pro^^ra m Background ' , " 

: ' » 

The heed for trained personnel in the area of environmental control has 
been emphasized by new governmental guidelines and requirements placed upon 
industrial operations in an' attempt to control and improve environmental 
conditions. In establishing need for the environmental training project, 
the project proposal stated: - - 
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There presently exists a^.alanning increase in the 
■pollution of our nation^s water, soil, and air,- 
which has made us aware of the critical manpower 
shortage at the professional and subprof essional 
level who are well trained and have special skills 
and knowledge to d^al with these problems in the 
^ field or in the laboratory (p. 2). 

The proposal noted that the western area of Kentucky has large coal 
fields, vast water impoundments ( and . Lakes)., in- addi- 

tion to local and/or regional' pro j ects such as watersheds, mines, and solid 
waste disppsal, which made it an ideal location for ^ environmental training. 
The proposal identified a need for persons to be employed as an Environ- 
mental Aide to assist the various agencies that had plans to expand their 
control and operation in the area' of environmental control. Also the- pro- 
posal established that various governmental agencies already involved in 
environmental control in the geographic area were willing to lend assist- 
ance for the development of course materials and for affilia-tion of trainees 
in a cooperative training project. ^ 

In response to needs identified- above, the Regional Director of Voca- 
tional Education (Mr. ) held a meer.ing with local agencies to 

determine if an Environmental Aide training program was needed and feasible;. 
As an .outcome of the meeting, a^ special committee was formed to directly 
involve the agencies that would participate in the development of the 
project, along with state, regional, and local* consultants . Subsequent 
meetings resulted in the establishment of criteria for occupational needs 
and project development. ^ - 

The committee identified tests, procedures, techniques, and theory 
needed to conduct the program and the facilities, equipment, and materials 
required. The curriculum develc^ped wa:s -designed to' be .relevant to the needs 
of participating agencies. In addition, the agencies involved further 
agreed to particiDate in the pro'gram by making, available special materials, 
instruction, and on-the-job work experience for students. ' . . • 

Geographical boundaries for the program and the rationale for their 
establishment were described in project documents: ' 



The area includes the region west of a'Mne from 
Louisville , south to Elizabethtown, to Bowling 
Green, and to the Tennessee border\^ It was de- 
cided that this area would prevent ^ over satu- 
ration of students compared to jobs. \Also, by 
covering an area -this large, we could include a ' 
more diversified kind of program due to aM the 
different kinds of process in the area (p.\2) . ^ 

Further information regarding maj or areas of. potential employment for 
environmental aides were identified in the document. These included posi- 
tions in: the Soil Conservation Service y a governmental agency which had 
a classification for employees on the assistant level; County \Eeatth 
Department y located in., each of the 40 counties included in theNgeographical 
boundaries of the program; and Sewage and Water Treatment PlantsY^ located 
in the 75 municipalities of the program. ^ \ ' ' 
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A project proposal for Environmental Aide training was prepared and 
submitted to the U. S.. Department of Health, Education and. Welfare on 
August 8, 1970, Approval was granted and the program became operational 
in the early part of May 1971. ' ' 

Instructional Program \ 

The Environmental' Aide program is one ^of 44 vocational training pro- 
grams serving over 9,000 post-high school students in Vocational Region 2. 

( ) of Kentucky. Glass enrollment is limited to 18-20 students 

from Vocational Regions 1-5. At the present time, there is an open-entry, 
open-exit policy in the class to accommodate early completors and allow them 
to be replaced by other persons who wish to enter the program; Thus far, 
52 students have trained in the program; approximately 81 percent of enrollees 
have been successful completors. 

Instruction includes formal classroom teaching, programmed instruction, 
multimedia usage, small^roup discussion, individualized instruction, prac- 
tical laboratory/w^-k^fiop experience, and supervised on-the-job experience. 
Of these, individualized instruction, pi^r ticularly with the open-gentry, open- 
exit program policy, is considered to be the most important. 

Various" types of evaluation techniques were utilized to' measure student 
achievement. These included paper-and-pencil tests (instructor-developed 
in addition to commercially prepared tes ts) , overall subjective rating by the 
instructor, instructor ratings of performance and/or products made during 
the course, performance examinations, oral examinations, and evaluation by 
supervisory personnel in the job affiliation phase of the program. The 
evaluations received by trainees in^ their job affiliations were considered 
most important. ■ . - . ; 

Program Evaluation j ' 

Evaluation Background 

' ' - •■ . 

The Environmental Aide training program was evaluated in 1972, 1973, 
and 1974. At the time of the interview, general information was available - 
concerning the three annual evaluations which were conducted and specific 
information was available for the 1973-74 evaluation in a "final report. 

According to governmental requirements, contractual provisions for ^evalua- 
tion -were* included in the project proposal as, follows: • 

A brief written eva'luation of the project which 
, includes assessment of: local administration, 
instruction, supervision, trainee achievement 
and placement',' and recommendations for improving 
the project will b-a submitted, to the. State agency 
by the local supervisor within 30 days after the 
completion of the project or at such other times . 
as the State may request (p. 8). 
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Evaluations Conduc ted 

The yearly evaluations, conducted foTx^the Environm^tal Aide program . 
which were all the'i same utilized four MDTA-developed instruments: MDTA 
Former Stucdent Follow-Up, Employers Follow-Up of MDTA Trainees, MDTA 
Instriictoy ' s Project Evaluation, and MDTA Supervisor 's Project Evaluation. 
The final evaluation report prepared for the proj ect . included tabled data 
from the follow-up mail surveys of former" trainees and their employers plus 
a narrative description^ of the data and the instructor's and supervisor's 
responses to the two MDTA evaluation questionnaires. More detailed infor- 
mation about the evaluation will be provided in a later section. 

Evaluation Model, Goals, Values 

Model. Although no model for the evaluation conducted was formalized, 
the evaluation could perhaps be roughly described as discrepancy evaluation. 
The information sought in the-MDTA ins truments sought to ascertain the 
extent to which the program was fulfilling its function—that is, training " 
students to function effectively as envirorjnental aides and providing 
environmental control agencies with trained personnel. 

The goals of the evaluation were to measure program impact and yto 
improve program effectiveness. The follow-up mail survey of former / trainees 
"and their employers attempted to measure program impact; all instruments 
used in the evaluation contained items which sought to identify specific 
aspects of the program which could be improved. 

The evaluation yielded both hard data concerning the .employment of 
trainees (ntimber employed, type of employment, job duties, and salary 
received) and .subjective data utilizing the opinions and judgments of 
former trainees, employers of former trainees, program instructor, and 
program supervisor on a number of program variables. 

Values* The evalus tiqn, conducted was intended to fulfill the con- 
tractual agreement included in the project proposal for a "brief written 
evaluation of local administration, instruction, supervision, trained 
achievement and placement, and recommendati^ons for improving the p^oject.'''^ 
Utilizing MDTA evaluation ins truments the evaluation sought to involve | 
various categories of persons who could provide diff erent ' perspectives j 
concerning the implementation and effectiveness of ^the 'program. v ~ ^ 

Evaluators . ^ " 

The evaluation was ■ conducted by ^the program instructor and program ^ 
coordinator as part of their job duties.* They were selected to perform 
the evaluation because- of their involvement with the program. Their eval- 
uation activities consisted of completing the MDTA instructor and super- 
visor evaluation questionnaires, conducting the follow-up mail survey of ^ , 
former trainees and' their employers, preparing tables, and describing ^ / 

findings, and assembling the final document. ^ » / 
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The backgrounds of the program instructor and coordinator were voQa- 
tionally rather than research-oriented. The instructor had a B. S. degree 
in agricultural sciences and six years of program-related V70rk experience. 
The coordinator had a B,S.. degree in indus trial, education and three ^years" 
of occupational experience. ■ " ^ * ' • 

Funding. Evaluation costs were internally . funded . Provisions for 
evaluation were specified in the proposal and were considered part of the 
project budget costs. * i ;* \ ■ . 

Evaluation Audiences ' ' * 

There were a variety of aud'ierices of the evaluation of the^ Environmental 
Aide training project. These includeii personnel from the Department of ^ ^ - 
Health, Education and fJelfare; the State Bureau of Vocational Education 
including the' Division of Vocational Program Development and the Division 
of Interagency Relations; and the Director of ^ Vocational Region 2. .Other 
audiences from- education inclitffed the principals of shop- vocationa:l schools, 
other -local school principals, and ^the regional program coordinator. Pro- 
gram trainees and affiliating agencies were other concerned audiences 'of 
the* evaluation. ^ ^ , , 

Expected Evaluation and ' Out;comes ^ ■ . 

If was anticipated that the evaluation would provide relevant feedback 
concerning tljie employment o£ the former trainees. ,0f par ticular interes t 
was the type/ of environmental control agencies at which the former employees 
were employed the level at which they functioned, and how Well they pef- 
formedon the job. It was also anticipated that the evaluation would pro- 
vide guidelines >f or curriculum impr'ovement ; ■ ■ . . 

Evalua ^^^n Descriptions ■ - 

• The evaluations conducte^i^were id^ntifi^d in an earlier part of this 
report. Ajmore detailed description follo\^rs : " ■ 

1. Follbw-Up Survey.' A mail questionnaire survey of former students 
and their 4mplpyers was- conducted, .as required by >DTA regulations. ^The^ 
student survey which consisted of four .mailings conducted at .two-week 
intervals was conducted one month after the end -of class. The follow-up 
of employ.ers consisted of only one mailing which was conducted six weeks 
after cla'ss had ended. Responses to' the survey were tallied and statistical 

^ tables were produced. A narrative explanation -of the tables was then 
prepared . ' v ^ 

2. . Supervisor Project Evaluation. -The MDTA evaluation form^ re- , 
quested data concerning project enrollment and -included 19 questions 
addressed to ""the project supervisor, eliciting both ^informati*on and sub- 
jective judgment. The items jLela.t-ed t-o^-progr am implementation, instruc- 
tional practices^,^.dequacy'"'of facilities-, equipment "and supplies, student 
services of"fered, and suggestions for program improvement. ' . 
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3. InstviLctov Project Evaluation, Sixteen questions to be^ answered, o 
by the ius.tructor were included on this MDTA form.. The Items, which elicited 
both information and subjective judgment, related .to trainin^g obj.ectives, 
instructional methods and techniques, training ^ajlds , adequacy '.of facilities , 
equipment and supplies,, student evaluation, and pro'j'ect improvement^* ' 

4. Final Evaluation Report. The final evaluation report did not in- 
clude 'any formal cohclusions section. However, several conclusions concern- 
ing various program aspects were recorded in' the instructor and -sup.ervisor 
-evaluation- forms : (1) the high rate of job placement was the greatest 
strength of the program; (2) the Section on sorl conservation should be 
omitted from the curriculum due to- theHlack. of available jobs in that area; 
and (3)' classroom supervision should be arranged during the job affiliation 
phase of the program, while the .instructor was engaged in agency visitations. 

Technical . Competence ' " ' * - 

As noted previously, the instruments and procedures used in the eval- 
uation of the Environmental Aide training program were reqCiired under-MDTA 
guidelines. The .evaluation was essentially a descriptive study of'^the pro-- 
gram and its impact. Evaluative judgments we,re based on the subjective 

opinions of' program instructor and sjipervisor. ' - 

' • ^ 'J • 

The variables measured by the MDTA instruments had 'obvious relevance 
to the purpose of the evaluation. Theformer student "follqw-up question- 
naire, however, did not include items that should have provided additional 
useful information, such as: how often on the job did former trainees use 
the knowledge and skills acquired in the. program; how related to tjhe job 
was the training received, and how^ long had\ students served in their jobs.. 
Also, no follow-up of dropouts was conducted. The evaluation report, itself 
consisted only of an assemblage of documents and did not include^ any inter- 
pretation of 'the findings. • 



Accomplishment of Goals 

The Director of Vocational Region 2 expressed satisfaction .with the 
evaluation and considered that.it. had accomplished its goals. The evaluation 
providfcid data on job placement of trainees, information on their employers, 
and other program and job related' information.' Thej^evaluation also identi- 
fied /program and curriculum areas for program improvement. 

t, ■ . 

Usefulness of Evaluation \^ 

The high rate of trainee placement di.sclos^eii' by the evaluation helped 
provide assurance to tho'^se closely involved with the program (coordjinator , 
instructor, trainees, and job affiliation agejicies) that the program^ was on 
the right track. The evaluation did lead to changes in curriculum, 
instruction, and program enrollment practices. ^ ' . . 

*>) ' • " ■ 

The curriculum unit on soil conservation was drastically reduced be- 

pause oT. lack of jobs in that area. In addition, an adjustment was made 

in the allotment of program hours — more time* was .apportioned tq^ the job 
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affiliacion pha3e and less to classrdo.iQ/laboratoi-y ,,wprk. Also, trainees 
no latxger had to spend a year in th^ program.. They could be awarded a 
certificate* of achievement based on instructor judgment, of their compe- 
tence at: any t^'e during the program. This open-entry, open-exijt policy 
made it /possible to^ accommodate more trainees and to increase the cost- 
effectiveness of the program. 

^ " A ■ ' ' ' ' . ' 

Unintended'^ Outcomes ' ^ • • 



The "evaluation did- not- focus on any program side effects. Follow-up 

of the program dropouts mi,ght have yielded useful information on side 

effects^ The district is planning to do additional follow-ups of former 

trainees one" three, and five years after program completion. ' ■ 

* , ■ » ♦ 

Positive Aspects of Evaluation ' ' • 

/ . ■* • • . 

The mos t, note:^i7or thy aspect of the evaluation was its inj5ut of different 
points of view. These included employers, former students, the instructor, 
and the supervisor. , ' 

Evaluation Problems ' ' \ " 



Several problems concerning the evaluation w-ere identified by Mr. 



the Director of Vocational Region 2. A higher rate of return of survey^ ' 
questionnaires might have -been achieved if further attempts .had been made 
tb £race and follow7up nonrespondents ., Also, the -follow-up of former trainees 
ancl'-their employers conducted four and six weeks, after program completion) was 
too* short a period df time foxP valid"appraisals to be made. However, the!^ 
timing of follow-up wa3 specified by gpvernmental regulations . As noted 
^ above, the district plan3 to do three additional follow-ups — one year, three 
years > and five years af'ter program completion,., " , . ■ 

Special Lessdns to be Learned' from Studying Evaluation - 

The most obvio' s lesson that can be learned from this evaluation is 
the import^ince of its role in measuring program ^impact and in identify:yig ^ , 
'areas for program improvement. Other lessons that can be learned are the 
importance of^ effective timing in conducting a follow-up; establishing ^> 
procedures for keeping up-to-date mailing addresses of trainees; and 
instituting different types of ■ follow-up procedures for nonreWponders . ' 
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REPORT OF SITE VISIT TO TECHNICAL INSTITUTE, _' 

WISCONSIN CONCERNING A PLASTICS TECHNOLOGY PROGRAM 

Getleral Description - • • , 

' The plastics Technology program off^ered at the ' Technical 

Ijistitute- in , Wisconsin, is a relatively new program which has 

received accreditation from the Wisconsin Board of Vocational, Technical 
and Adult Education. The ^rogruf^ which became operational in the. .1971- 
72 school year is a two-year technical training program leading to an 
Ass'ociate Degree. , ^ • 

The program provides enrollees with a comprehensive background i'n 
the techni'cali ties of plastics as well as the fundamentals of operation. 
In addition to the courses in -the theoreticaL and practical aspects of 
plastics, the curriculum includes courses in general education as pre- 
•scri^ed by 'the Wisconsin Board of Vocational, Technical and Adult Education. 
These include technical mathematics-, technical science, communication skills 
psychology, of human relations, methods of supervision, economics; and 
American Institutions. \^ ' • 

District 11 is the only^ VTAE district in the State to 

^ offer an, associate degree program in plastics technology. The program ^ 

is offered only at the ' Technical Institute' in _^ which 

has an open admissions policy for entry into the. program. Enrollees are 
generally required to have a high school diplpma or. its equivalent. All 
enrollees pay registration and special course. fees. ^ Tuition is free for 
District 11 and other state residents; non-state residents are charged 
tuition to cover instruction costs. , 

■Enrollees consist primarily^^ of students residing within District 11. 

The 'district* presently comprises two counties-- _^ and , 

each of:.whose population was reported over 80,000^in 1970. The populations 
of bqth counties have shown a steady increase over the past decade and 

their projected population by 1980 is 87., 722 for County and ^ 

93,957 for County. ' 

Approxim'ately 4'0 percent of the student body- at the . Tech- 
nical Institute in ^ are-recent high school graduates, and approx- . 

imately *60 percent are employed adults or veterans. The average age of 
the students which has shown a steady increase is now about 22 years. The 
Institute has a tota:l enrollment of 1,344 FTEs. (A' 'full-time equivalent 
(FIE) student is one who ^is taking 15 credits per semester,) At th^ time' 
of the interview, it was reported that there were 31 FTEs in the Plastics 
Technology program. Of these, 26 were full-time students and the others, 
attended on.- a ^part-time basis. 

Program Background ^ - 

Impetus for the program began in the late '1960s when a nationwide 
surv^ey of over 4,000 plastics processing firms disclosed a serious short- 
age of trained personnel. The survey, conducted by a joint committee 
of. the Society of Plastic Engineers (SPE) and the Society of Plastic 
Industries (S'PI) in. 1967, found the shortage to exist in direct proportion 
to the geographical locations of plastics processing plants. 
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A large, number of plastics processing plants are located in 
Wisconsin, particularly in Lne southeastern area of the state. Within 
District 11, there are approximately 5,000 persons employed in the 
plastics processing industry. There are five major companies in the 
district and many peripheral industries such as those which manufacture 
electrical control devices, packaging products, bathroom accessories, 
and floor tiles. 

/ 

The plastics industry, a re.latively new and rapidly developing 
industry had urged technical education in plastics over a period of 
years. An article appearing in the SPE professional journal in 1969 
particularly acknowledged "the effectiveness of twor-year training pro- 
grams. 

Local plastics industries within District 11 and * the surrounding 
areas made known their needs for trained personnel through the district's 
field services coordinators. A, state survey of, need was conducted by 
a mailed questionnaire survey of plastics processing firms' throughout 
the state. Information sought in the survey focused on the type of 
plastics operation by processes, number of employees, number of plants, 
present and future employee needs, type of trained personnel neede"d, 
and interest .in hiring technically trained persons. Results of the survey 
reaffirmed the need for trained personnel and reflected interest and 
support for a Plastics Technology program.' ■ . . ' | 

An ad hoc Plastics Technology Advisory Committee was formed to 
help establish a training program. The committee included representatives 
of. local plastic industries , District 11 administrative members, and an 
advisor from the State Board of Vocational, Technical and 'Adult Education 
( ) . The efforts of the Committee resulted in the identifica- 

tion of potential' job openings in the plastics industries, formulation of 
a relevant curriculum, and * specif icatibns for required physical facilities 
and equipment. Their proposal for a two-year associat-e degree program in 
Plastics Technology was submitted to the State Board of Vocational, Tech- 
nical and Adult Education in August, 1970, and was granted approval. 
Approximately one year later, , the program was- initiated. 

Instructional Program 

The Plastics Technology program "was designed to prepare people to 
enter the plastics industries at the technician level. The program is 
intended to help meet state 'and local employment needs for trained per- 
* sonnel and to provide students with the best possible education to help 
insure their emplo3r[nent in the plastics industry. 

The Institute provides prospective students in the Plastics Tech- 
nology program with a general job description of plastics technicians 
and their typical job duties, and a listing and sequence of courses 
included in the program curriculum. In contrast to one^year^^vocational 
programs; the technician level program places less emphasis on manipula- 
tive skills and more on theoretical concepts. ^ , 

Course descriptions and outlines have been developed for the courses 
in the program. Some of the courses have written behavioral objectives 
which are stated in the Institute's Monograph. Instruction in the program 
includes formal classroom teaching, individualized instruction, group 
discussion, multi-media usage , and practical laboratory/workshop e:<perience. 
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Various types of student evaluation are utilized to measure student 
'achievement: instructor developed paper-and-pencil tests containing 
objective items,, essay questions, and* questions involving problem •• 
solving;, the'^oyerall subjective judgment of the" instructor; and instructor 
ratings -of performance and products made during the course. Another 
evaluation method that is "used involves a simulated employment situation. 
Students are required to complete a project in plastics from its basic 
design through final production, incorporating their theoretical and 
practical knowledge of plastics. In addition to demonstrating their 
'technical skills, students display their communication skills by making 
an oral presentation of their project* to their peers in class. They 
receive instructor ratings on both product and presentation. 

Program Evaluation ' ' / ' 

^ Evaluation B^ackground . ^ 

Under State law, the Wisconsin Board of Vocational, Technical and 
Adult Education serves as an accrediting agency for associate degree 
programs at the technical institute level. Board policies, procedures, 
and regulations 'are documented in its publication, ^'Guidelines for 
Program Development and Evaluation — Vocational Diploma^ Associate ■ 
Degree,'' issued July 10, 1968. 

A major purpose of Board Accreditations as stated in the Guidelines 
has been...;, "to assure ... that standards adequate to achieve a suitable 
teichniQal education have been established by a legally constituted state 
administration agency "(p. 17)." The Guidelines stress that the process of 
state administration is one that, emphasizes direction through involvement 
rather tKan impdsed authority. Accordingly, district administrators and ■ 
other district representatives are used in advisory capacities and the 
published Guidelines* which specify evaluation as well as program develop- 
ment procedures were developed by groups composed of local school and 
district representatives- working with state staff. 

Evaluations Conducted ^ 

The evaluations conducted for the Plastics Technology program are 
standardized procedures used to evaluate the vocational and technical 
programs offered by" the-State VTAE districts. The evaluations of the 
program which were conducted are briefly described in this section. 
More detailed information is provided in later sections. 

1. Accreditation Evaluation. This was essentially a process eval-- 
u^.tion conducted at the .end of the second year of the technician level 
program. Its goals were to help establish accreditation for the program 
and to help improve the program.. The evaluation of the Plastics Tech- 
nology program^, which was internally funded, was carried out by the 
Wisconsin Board of Vocational, Technical and Adult Education and involved 
the utilization of • an external team of industry and education representa- 
tives. Output of the evaluation was a formal, evaluation document. 

■I . ■ 

I ■ ■ ■ 
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2. Impact Evaluation. This was essentially a product evaluation 
conducted to assess the impact of the program. Its focus was on student 
graduates of the program. It utilized a follow-up questionnaire designed 
by the State Board which was sent to students six months after program 
graduation. This evaluation, which was internally funded, was carried 
out by the district program developers; data processing was conducted at 
the. State Board. Output consisted of a computer, print-out sheet 
statistically sijinmarizing the results. Additional follow-ups of graduates 
are to be conducted 2 1/2 and 5 years after graduation. * 

< ■ ■ • 

3. Program Review* A standardized form, ^^Program Review and 
Developme.nt^^^ is- completed internally for each program at the district 
level. The form which is completed on an annual basis is used to provide 
statistical data on enrollments, staffing, and facilities, cost data, 
general informatipn, and placement of graduates and noncomple.tors of the 
program. Output consists of the completed document . 

The program review data is studied by representatives of ^the State 
Board and district administrative and instructional ..staff . The information 
is used to make decisions concerning program continuance and fiscal support 
This form was. completed for the Plastics Technology program following its 
second year of operation and annually thereafter. 

Evaluation Model, Goals, and Values 

Model. The type of evaluations conducted for the Plastics Technology 
program could perhaps best be described as 'discrepancy evaluation", which 
has been defined as follows: y ' . . 

Discrepancy evaluation refers to the search for 
differences between two or more elements of vari- 
ables of an education/ training program that, 
according to logical, rational or statistical 
criteria, should be in agreement or correspondance. 

A state survey of need was conducted to ascertain the discrepancy 
between the need and availability of trained technicians for the plastics 
industries. This survey provided the stimulus for the development of 
the Plastics Technology program. 

Also, program standards appearing in the published Guidelines have . 
been established by the State Board of Vocational, Technical and Adult 
Education., A major evaluation effort related to program accreditation 
and sought to ascertain the relationship between the established" standards 
and the actual program implementation. An external advisory committee 
including representatives of .the plastics industry 'and other educational 
institutions was utilized" to see that the program met the standards estab- 
lished by the Board and to identify discrepancies between actual and desir^ 
able- program aspects. The student follow-up was also intended to help- 
identify ..discrepanciesrbetween desirable anS actual program effects. Both" 
the impact and program review evaluations identified- previously were 4.^- 
tended to ascertain that the program was fulfilling its function according 
;to established program" goals and cost-effectiveness criteria.- ' 
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Goals. The goals of the evaluation were to establish program 
accreditation according to established standards and to identify program 
areas in need of improvement. In- addition ,. the evaluations sought to 
determine whether the program was fulfilling its function, that'is, pro-., 
viding the plastics industry with adequately trained personnel, and train/- 
ing students to function effectively as plastics technicians. 

Values./ Basically, the State Board sought , to involve .in the' eval- 
uations all levels of district and sdhool personnel, as well as attend- 
ing students and program graduates. Also, the Board made a. concerted' 
effort to obtain a broad perspective/of the program from the viewpoints 
of industry and education by -inviting a select group of representatives 
from these areas to serve on the evaluation committee. In addition, 
the accreditation evaluation was used to provide training for several 
new administi:ative' staf'f members . 

Evaluators 

The accreditation evaluation was initiated at the request of the 
program administrator to the State Board of Vocational, Technical, and 

Adult Education. Mr. who is the Board project consultant, 

assembled the team of external consultants to serve on the Associate 
Degree Evaluation Committee for the Plastics Technology program. 

Committee members were carefully selected so that they might pro- 
vide a broad and unbiased perspective of the"" program by plastics in- 
dustry and education representatives. The committee members included 
three persons from the, plastics industry, one staff member from the 
state university, one administrator of instructional services from an- 
other Wisconsin VTAE district, and a member of the Wisconsin Board of 
Vocational, Technical and Adult Education. • 

The impact evaluation, which involved the mailing of survey ques- 
tionnaires to program graduates, was conducted at the district level 
* (Instructional Services) ; data were processed at the Board and released 
in the form of a computer printout sheet. "'Data for the. program review 
study was .gathered by school personnel members sfo be reviewed by 
representatives, of the State Board, and district administrative and in- 
structional staff. " . 

Evaluation Funding 

Evaluations are considered to be part of the program and their 
cost^ are absorbed by the program budget.- The travel, room and board 
expenses of members of the Associate Degree Evaluat^ion Committee for 
the Plastics Technology was paid by the District. Committee members^ 
time costs were absorbed by their employers. 

Evaluation Audiences . ^ . 

There were a variety of .audiences of the evaluation of the Plastics 
Technology program. These consisted of staff members of the State Depart 
ment of 'Technical, Vocational and Adult Education who were involved in 
establishing program accreditation, District '11 and school administrative 
and instructional staff members involved with the program, and student 

' . 142 ' ^ . ■ - 

■ . D-56-' ■ ... 



enrolleeS\ who discussed the relative merits of the program with the 
committee members. Other persons in education observed the Plastics . 
Technology evaluation as an evaluation learning experience. Also, 
the Plastics Technology Advisory^ Committee/, a local program advisory 
committee comprised of people in the plastics industry, was a very 
concerned audience of the evaluations as was the State project . 
consultant. 

Expected Evaluation Outcomes and Benefits . . 

Program accreditation was a major expected outcome of the process 
evaluation.. In addition, it was anticipated that the committee would 
provide input for program improvement. It was also expected that the 
evaluations would help ascertain that the program effectively^ served 
the needs of industry and enrollees, and that it was making progress 
in meeting its -goals. 

Evaluation Descriptions 

The evaluations conducted were identified in an earlier section of 
this report: . * 

1/ Accveditation Evaluation. This evaluation was conducted ac- 
cording to standardized procedures established by the State Board of 
Vocational, Techi\ical and Adult Education. Following the request of . 
the program administrator for an evaluat-ion at the -end of the program's 
second year, the State Board consultant assembled an appropriate com- 
mittee to conduct the evaluation. • ' ' 

District- and Institute staff members performed a self-study which 
was reported in a monograph forwarded to the Committee members for 
study. Following review of the materials, the Committee made a three- 
day site visit during which time it performed the evaluation. Committee 
memb.ers received- briefing from State Boaxd members concerning the eval- 
uation and were provided with the '^Evaluation and Program survey Check- 
listj^' developed by the State Board. ... 

In performing < the evaluation. Committee members employed observa- 
tional techniques and conducted interviews with the administration. and 
instructional staff as well as with students, using the checklist as a 
guide. On its final report j. the Committee provided four different levels 
of reaction: commendations, suggestions, recommendations, and condition 
of approval. Responses of the committee members were recorded on tape 
and the transcriptions formed the basis of the final' evaluation document. 

The Committee made many commendations, suggestions, arid recommenda- 
tions, relating to the major categories listed in th% ^'Evaluation and 
Program Survey Checklist:'' students, faculty, curriculum and instruction, 
facilities — equipment and supplies, local advisory committee, and leader- 
ship. Concerning students, the Committee observed tl\at their attitudes 
were outstanding and that they were definitely motivated. " The Committee 
expressed concern, over the high attrition rate in the program (over 50 
percent) and suggested that a follow-up study be conducted 'of those who 
had dropped out. 
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The Committee commended the work and effort of "the instructors in 
establishing the program. Recommendations made concerning the technical 
faculty members were that they become more involved with industry and 
industrial practices, and that they participate more actively in profes- 
sional organizations. Various recommendations were made concerning cur- 
riculum and instruction. It was reco.mmended that learning objectives be 
established for all units and courses, that objectives be defined clear- 
ly, and that objectives be reviewed to ascertain that they were in line 
with realistic opportunities for student placement. It was recommended 
that emphasis be shifted in the economics and communications courses 
and that the supervision course be , dropped from the curriculum. The 
Committee also recommended that the Institute provide extension courses 
in plastics technology or evening and summer courses. 

Concerning instructional facilities, the Committee made recommenda- 
tions for better safety practices, changes in space arrangements, and 
the purchase of additional. equipment . It was also recommended that the 
local program advisory committee be enlarged to include -a greater cross- 
section of those involved in various levels of activity in ^the plastic 
industry — top management, laboratory, and shop employees; and that the 
advisory committee consider, the . re-evaluation of the program in' two years. 

2. Impact Evaluation* Results of the follow-up survey of the 
1972-.73 Plastics Technology program graduates were reported on a computer 
printout sheet. It showed 100 percent return of the questionnaires for 
the eight graduates for that school year. Of these, five were employed 
full-time in . an occupation related to their training and they received 

an average monthly rate of $602. Three respondents were employed in- the 
district where they were trained, one was employed-^out of the district 
but^ in-state, dnd one was employed out-of-state. Two respondents ob- 
tained their job during training and the other three, after training. 
None of the unemployed respondents were seeking employme^it.,. The majority 
of the respondents reported- that they felt the training was satisfactory. 

3. Program Review. Although the ^^Progrm Review and Development^^ 
forms had been reported completed for the Plastics Technology program, 
they were not available at the time of the interview. 

Technical Competence ' . ^ 

The procedures used in the evaluations of the Plastics , Technology 
program have been 'used operationally by the Wisconsin Board, of Vocation- 
al, ' Technical and Adult Educat ion since 1963. The evaluation measures 
used were appropriate to the questions raised in the evaluation and had 
observable content validity. 

f- ■ * - 

The ^'Evaluation and Program Survey Checklist^' throughly identified 
variables the State Board considered important in establishing standards 
for accreditation and program improvement. Evaluation ' findings were sub- 
jective in that they relied upon expert judgments to determine that the 
program met accreditation standards and to identify discrepancies between 
actual and desirable program aspects. 

The follow-up and program review studies provided statistical data 
which wa^ amenable to further analysis and interpretation by reviewers. 
'However, no formal evaluation document incorporated their findings. 
Further information concerning program effectiveness might have been 
sougl'il:. from employers, employees, part-time students, and dropouts. It 
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is anticipated that future evaluations will cover these areas. 

Accomplishment of Goals 

The State Board of Vocational ^ Technical and Adult Education con- 
sidered that to a great exterit'^ the £oals of the evaluation had been ac- 
complished. The program received accreditation and. the team of experts 
provided many suggestions and recommendations for program improvement. 
Most iof the administrative., and instructional staff were pleased with 
the evaluation and found the suggestions and recommendations to be use- 
ful to them. The District Board, Advisory Committee, and attending 
students also" expressed satisfaction with the evaluation. 

Inasmuch as there were only eight graduates of .the Plastic Tech- 
nology program in 1972-73 and the follow-up was conducted only six 
months following graduation, it was not possible to objectively ascer- 
tain whether the program was providing the plastics industry witsh well- 
trained personnel. , 

Usefulness of E'C^'aluation 

The evaluation results' were considered most useful by the admini- 
strative and instructional staff. The program was' expanded to the 
evening sessions to include more courses in plastics for those who 
worked during the day. Some new machinery and equipment was. obtained. 
The instructors became active in the professional plastic associations 
and did summer work in the plastics industry; more work was done on 
preparation of instructional material. 

The evaluation also provided for follow-up. One of the evaluation 
suggestions was that the local Plastics Advisory Committee consider the 
re-evaluation of the program in two years. At the time of this inter- 
view the re-evaluation of the curriculum was being comp'leted. 

Unintended Outcomes 

f . • ■■ 

Concern for high attritio.n rate of student enrollees was ex- 
pressed in* the evaluation. It had been projected that a student enroll- 
ment of 80 PTEs was required to make the program '*pay off^. As noted 
previously, at the time of the interview, there were 36 PTEs in the 
program of whom 26 were full-time students and the others, part-time. 
The evaluatioil suggested that a follow-up study be conducted by the 
local' administrative staff to determine what happened to"the. students' 
who. dropped and why they did so. 

An hypothesis off ered was that students may have been 'Vork-outs'!; 
rather than dropouts. That is, they may have left the program "to get^a 
job in the plastics industry. Two of the five employed 1972-73 graduates 
reported that they became employed in the plastics^ industry while still 
attending school. The evaluation committee considered that' the program__^ 
could be used on a. part-time basis for adults employed in the p.lastlcis 



industry. They recommended that the program' of fer 'courses to accomodate 
part-timers. . This was followed through with programs being offered in 
the evening session. 

Positive Aspects of Evaluation 

The involvement of many individuals reflecting different points of 
view in the evaluation .was noteworthy . " These included views of those 
in industry , education, State Board., administrative and instructional 
staff, and attending students. The evaluation gave the institution 
some ass^irance that they were on the "right track," placed renewed focus 
on the importance of instructional goals and objectives, and identified 
the need of specific facilities and equipment. ' . 

The evaluation involved local plastics industry people in curric- 
ulum planning. If helped ^assure the industry^of the quality of education 
being offered, and through the evaluation team, identified potential 
employment possibilities for students and graduates. 

Although, the opinion was offere'd that more time would have been 
helpful in performing the evaluation, the speed and organization with 
which the accreditation evaluation was accomplished was most impressive. 

Evaluation Problems ... 

Although the smooth organization made it 'possible to conduct the 
actual on-site evaluation in three days J additional time would have 
permitted the team to have been more thorough and to provide greater 
indepth evaluation. Also,^ use of a recorder in the evaluation presented 
various technical problems. 

The self-study as presented^ in the Monograph was not organized as 
well as it might have been. It was considered that its effectiveness 
would have been increased had it been more complete concerning behavioral 
objectives, more concise, and had eliminated irrelevant sections. 

Special Lessons To Be Learned 

The'most obvious lesson that can be learned from this evaluation 
is the;- contribution that evaluation can make in hel-pl^ng assure an 
effeC'tive training program. The invo^^men't'-'of^ndustry staff members 
in the evaluation of a technical-^traini-ng program appears to be of 
major importance in acjaompTishing the program's goals. 



^JRa-rticularly noteworthy in the evaluation were the comprehensive 
^program variables identified in the evaluation instruxuents which may 
be highly useful to other program developers , administrators , and in- 
structors. The techniques used to orient the evaluation team also ap- 
pear to be a highly effective approach in conducting such an evaluation, 
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*Millington, TN 38054 
ATTN: Dr. Normal J. Kerrv 

Cpramanding Officer 
Service School Command 
U.S. Naval Training Center 
San Diego, yCA 92133 
ATT51: Code 3030 

Principal Civilian Advisor 
• for Education and Training 
Naval Training Command, Code OOA 
Pensacola/'FL 32508 
ATTN: Dr. William L. Maloy 



LCDR C.vF.^ Logan, UNS 

F-14 Management System 

COMFITAEWINGPAC 

NAS Miramar, CA 92145 

Navy Personnel Research 

and Development Center 
Code 01 

San JDiego, CA 92152 

Navy Personnel Research;*^' 

and Development Center 
Code 02 

San Diego; CA 92152 
ATTN:' A. A. Sjoholm 

Navy Personnel Research 

and Development Center. 
Code 304 

San Diego, CA 92152 
ATTN: Dr. John Ford 

Navy Personnel. Research 

and Development Center 
Code 306 

San Diego, CA 92152 

ATTN: Dr. J. H. Steinemann 

Navy Personnel Resear.ch 

and Development Center ^ 
San Diego, CA 92152 
ATTN:" Library 

D^. M. Gragg, CAPT, MC, .USJ!T. 
Head, Educational Programs 

Developmen-t- Department 
Naval Health Sciences Education 
and Training Cocnmand 
. Bethesda, I'lD 20014 
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Director 

Training Analysis S Evaluation Group 
Code N-OOt- 

Department of the \^s^rJ 

Orlando, FL 32813 _ \^ 

ATTN: Dr. Alfred F. Smode 7' 

Chief of :iJaval Training Support 
Code N--2r 
Building 45 

Naval Air Station ^ ., . 
Pensacola, FL 32508 
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Technical Director' 

U.S.- Army Research Institute for the 
.y'Behavioral and S-ocial Sciences 
1300 Wilson Boulevard 
Arlington/ VA 22209 

Armed Forces Staff College 
Norfolk,' VA 23511:..' 
ATTN: Library 
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1 • Commandant ^, 

U.S. Army Infantry School 
Fort Benning, GA 31905 
ATTN: ATSH-DET 



1 Deputy Commander , ' ■■ 

U.S. Army Institute'^of Administration 
Fort Benjamin Harrison, IN 46216 
ATTN: EA . . . 

1 Dr. Frank J. Harris 

U. S.' Army "Research Institute for the 

.fiVhavioral §^d Social Sciences 
■1300 Wilson Boulevard 
Arlington, JA 22209 

1 Dr. Stanley L. Cohen 

U.S. Army Research Institute for the. 
Behavioral, and Social Sciences 
^ 1300. Wilson Boulevard 
Arlington, VA 22209 

1 l)r. Ralph Dusek 

U.S. Airmy Research Institute for the 

Behavioral and Social Sciences 
1300 Wilson ';Boulevard 
Arlington, VA 22209 

1 Dr. Leon H. Navnrocki 

U»S. Army Research Ii^atitute for the 

Behavioral and Socii3?^|:iences ' 
1300 Wilson Boulevard 
Arlington, VA 22209 

1 Dr. Joseph Ward 

U.S. Army Research Institute for the 

Behavioral and Social Sciences 
1300 Wilson Boulevard 
Arlington, VA 2.2209 

1 HW USAREUR''& 7th Army* 

ODCSOPS ' 
USAREUR Director of GEO 
APO New York 09403 

1' ^ARI Field Unit - Leavenworth \ 
Post Office Box 3122 
.Fort Leavenworth, KS 66027 

1 ♦ Mr. James Baker 

U.S. Army Research Institute for the 

Behavioral -and Social Sciences 
1300 -Wilson Boulevard • 
Arlington, VA 22209 



Air Force 

1 Research Branch 
- ^ ' AF/DPMYAR 

Randolph AFB, TX 78148 

1 Dr.^G. A. Eckstrand (AFHRL/AST) 
' . Wright-Patterson AFB 
Ohio 43433- ' 

1 Dr. Ross L. Morgan (APHRL/ASR) 
Wright-Pattferson AFB 
Ohio 43433 

.1 AFHRL/.DOJN * - * 

Stop //63 

Lackland AFB,.TX 78236 

' 1 Dr. Martin Rockway (AFHRL/TT) 
Lowry AFB * . 
Colorado 80230 

1 Major P. J,. ,DeLeo 

Instructional Technology Branch 
AF Human Resources Laboratory 
Lowry AFB, CO 80230 

1 Dr. 'Alfred R. Fregly • ' , " - 
^' AFOSR/NL 

1400"^ Wilson Boulevard 
Arlington, VA 22209* 

1 Dr, Sylvia R. Mayer (MCIT) 

Headquarters Electronic Systems 

Division 
LG Hanscom Field 
r . .Bedford, MA 01730 

1 Capt. Jack Thorpe, USAF 
, Flying Training Division 

'afhrl/ft 

Williams AFB, AZ 85224 

1 AFHRL/P'ED- 
.. Stop #63 
Lackland AFB, TX 7^236 - 
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Marine Corps 

1 Director, Office qf Manpower ! 
Utilization 
Headquarters, Marine Corps (Code MPU) 
MCB • (Building 2009) 
. Quant ico, VA 22134 

1 Dr. A, L. Slafkosky 

Scienctific Advisor (Code RD~1> 
dfea^iuarters, U.S. Marine Corps 
J;i.aington, DC 20380 

1 Chief, Academic Department 
Education 'Center 
Marine Corps Development and 

Education Command 
'Marine Corps Base 
' Quantico, VA 22134 ' 
*> 

1 Mr.' E. A. Dover . ' y 

2711 South Veitch Street 
Arlington, VA 22206 

Coast Guard 

-J. 'Mr. Joseph J. Cowan, Chief 

Psychological' Research Branch (G-P-l/62 
U.S. Coast Guard Headquarters 
Washington, DC 20590 • 

. ■ . ■ ■ i ^ 

Other DOD ' 



Military .'Assistant for Huiaan Resources 
Office of the Secretary of Defense 
Room 3D129, Pentagon 
Washington, DC .-20301 
• - . i • 
Advanced' Research Projects Agency 
Administrative Services' 
1400 Wilson Boulevard 
Arlington, VA 22209 ' ' 

ATTN; Ardella Holloway 
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J)r1 Harold- F. O'Neil, Jr.^ 
Advanced Research Projects Agency 
Human Resources Research Office 
1400 Wilson Bpulevaxd 
Arlington, VA 22209 . 



\ 

1 Dr.^ Robert Young ' " » , 

Advanced Research Projects Agency 
Human'' Resources 4lesearch Office- 
1400 Wi^lson Boulevard 
ArliligtQn, VA 22209 

1. Mr. Frede^rick W. Suffa 

Chief, Recruiting and Retention . , 

Evalu^atibn 
Office of the Assistant Secretary 

of Defense, M"&RA 
Room 3D970, Pentagon 
Washington, *DC 20301 ' 

■ d» ■ . 

12 Defense Documentation Center 
Cameron Station, Building 5 
. .Alexandria, VA \223l4 
ATTN:' TC " \ 

' ' U . ■ I 

0 ther Government \ 

■ t 
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1 ' Dr. William Gorham, Director 

Personnel Research and Developm^rit 
Center 

U.S. Civir Service Commission 
.1900 E Street, N.W.- 
Washington, DC 20415- 

Dr. Erik McWilliams, Director 
Technological Innovations in 

' Education Group 
National Science Foundation 
'1800 G Street,' N.W/, -Roonl W65Q- 
Washiilgton, DC 20550 

Dr; Richard C. Atkinson -^^ 
Deguty Dir.ector * ^ 

Naci<Sial Science Foundation ' 
ISOO^G Street, N.Wi - ' \ 
.Was'^hington, DC 20550 

•Dr. Andrew R. Molnar 
Technological Innovations in 
Education Group 
^ National Science Foundation 

1800 G. Street, N.W. 
' Washington, DC 20550 • 

Dr. Marshall S. Smith / 
Assistant Actinfg Director / 
Program on Essential Skills 
National Institute of Ed4c^tion 
Brown Building, Room 815 
19th and M Streets, N.W. 
Washington, DC * 20208 * 



) 



141 



Htscellaneous 

1: Dr^' John Annett 

"Department: of Psychology 
The University of Warwick 
Covencry CV47Ma 

1 Dr* Ronald P. Carver 
School of Education 
University of Missouri-Kansas City 
5100 Rockhill Road 
ICansas City, MO 64110 

1 ' Century Research Corporation 
' 4113 Lee Highway 
Arlington, . VA - 22207 

1 . Dr. Kenneth E, Clark 
University of Rochester 
College of Arts and Sciences 
/ River Campus St anion 
Rochester, NY 146£7 

1 Dr. Allan Collins 

BuJLt Beianek and Ne^-raaan, Inc. 
50 Hdulton Street ^ 
" ^ Cambridge, MA 02138 • 

1 Dr* Ruth Dajr 
Yale University 
Department of Psychology 
2 Hillhouse Avenue 
New Haven, CT 06520 

1 ERIC 

Processing rinrl Pvef*=^ronce Facility 
4833 Rugby Avenue 
Bethesda, MD 20014 

1 Dr* Victor Fields \ 
Montgomery- College \ 
Department of Psychology 
RcckviMe, MD 20850 

1 Dr* Edwin A* Fleishman 
Visiting Professor 
University of California 
Graduate School of Administration 
Irvine, CA 92664. 



Dr. Robert Glaser, Co-Director 
University of Pittsburgh " 
3939 O'Hara Street 
Pittsburgh, PA 15213 " 

Dr. M. D. Havron" . 
Human Sciences Research, Inc. 
7710 Old Spring House Road 
West Gate Industrial Park 
McLean, VA 22101 

HumPRO Central Division . . • 

400 Plaza Building \ 
Pace 'Boulevard at Fairfield Drive . 
Pensacola, FL 32505 \ 

HumPRO/ Western Divisio^ 
27857 Berwick .Drive 
^ Carmel, CA 93921 ^ . ^ 

ATTN: Library 1" , 

HumPRO Central Division/Columbus Off 
Suite. 23, 2601 Cross Country Drive 
.Columbus,' GA 31906 ' . ' 

HumPRO/Western- Division 
27857 Berwick Drive 
Carmel, CA . 93921^ 
ATTN: Dr. Robert" Vineberg, 

HumPRO 

oseph. A. Austin Building 
939 Goldsmith Lane 
sville, 'kY 40213 




Dr. Lawrence B. Johnso.n 
Lawrence Johnson & Associates, Inc-, 
2001^S' Street, N.W.,' Suite 502 
Washington, DC 20009 

Dr. Roger A. Kaufman 

U.S. International University 

Graduate School of Human Behavior 

Elliott Campus ^ 

8655' E. Pomerada Road 

San Diego, CA 92124 

Dr. Alma E. Lant2^> • 
University of Denver 
Denver 'Research Institute 
Industrial Economics Division • 
Denver, 'CO 80210 
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Dr. Robert R. Mackie 

'Human Factors Research, Inc.- 

6780 Cdrton Drive 

Santa Barbara Research Park 

Goleta, CA 93017 : 

Dr. Leo Munday, Vice President- 
American College Testing Prograirt 
P.O. Box 168 
Iowa City, IA\ 52240 

Dr. Donald A., Norman 
University of California, San^ Diego 
Department of Psychology ^' 
Lajt^llk, GA 92037 



Mr. Luigi Petrullp 
2431 North Edgewood Street 
Arlin^^ton, VA 22207 

Dr. Diane M. Ramsey-Klee 
.R^K Research &• System Design 
3947 Ridgemont Drive 
Malibu, CA- 90265 v ■ 

Dr. Joseph W. Ri^TiHy" 

University of Southern California 

Behavioral Technology Laboratories 

3717 South Grand 

Los Angeles, CA- 90007 

Dr. Leonard L. Rosenbauni, Chairman 
Montgomery College ; 
Department of Psychology 
Rbckville, MD 20850 ^ 



1 'Dr. Ratrick Suppes 
Stanford University 
■ Institute for Mathematical Studies 
in the Social Sciences 
Stanford, GA 94305 

1 Dr. Benton J. Underwood 
Nor: thwes tern -University 
Department .of Psychology 
Evans ton, IL .60201 

1 Dr. Carl R. Vest, •• 

' Battelle Memorial Institute 

Washington Operations 

2030 M Street, N.W. 

Washington, DC 20036 

1 Dr. David J. Weiss ' 

University of Minnesota 
Department of Psychology 
N660 Elliott Hall 
Minneapolis, M 55455 

1 Dr . K. Wescourt 

Stanford University ^ 

Institute for Mathematical Studies 

in the .Social Sciences 
Stanford, CA 94305 

1 Dr. Anita' West 

Denver Research Institute 
University of Denver 
Denver, CO ^ 80210 



Dr. George E. Rowland \ 
Rowland and Company, Inc. ^ 
P.O. Box 61 

Haddonfield, NJ 08033 

Dr* Arthur I. Siegel 
Applied Psychological Services 
404. East Lancaster Avenue 
Wayne, PA 19087 

Dr. -Richard Snow 
Stanford University 
School of Education 
"Stanford, CA 94305 



\ 



152 



143 



